Large scale integration of photovoltaic (PV) energy into power grids has seen an expressive growth. But PV power output variability hinders the large scale deployment of PV. Images captured from ground based sky imaging systems are commonly used in irradiance forecasting to address the intermittent energy production from solar panels. Facing that scenario, this study presents a novel approach to model the Global Horizontal Irradiance (GHI) based on contemporaneous hemispherical sky images (nowcast). A physics-based nonparametric classification model based on threshold of fractional cloudiness of sky images was utilized to classify the images into 3 sky conditions: sunny, partially cloudy and overcast. Several ResNet architectures were first examined as end-to-end models and then they were tailored to each sky-condition to create sky-condition based sub models. For each ResNet architecture, results of the end-to-end model and sky-condition specific sub models were compared. These sky-condition specific sub models were then combined to create a classification-nowcast framework. The best performing model yielded a MAE of 23.86 W/m 2 , nRMSE of 10.80% and reached 98.88% in Pearson’s correlation on the test set. The errors were comparable to or lower than other studies despite the high fluctuations in the GHI.