notebooks/spacenet/spacenet_rio_chip_class_data_prep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SpaceNet Rio Chip Classification Data Prep\n",
    "\n",
    "This notebook prepares data for training a chip classification model on the Rio SpaceNet dataset.\n",
    "\n",
    "* Set `raw_uri` to the local or S3 directory containing the raw dataset.\n",
    "* Set `processed_uri` to a local or S3 directory (you can write to), which will store the processed data generated by this notebook.\n",
    "\n",
    "This is all you will need to do in order to run this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_uri = 's3://spacenet-dataset/'\n",
    "# processed_uri = 's3://raster-vision-lf-dev/examples/spacenet/rio/processed-data/'\n",
    "processed_uri = '/opt/data/examples/spacenet/rio/processed-data'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The steps we'll take to make the data are as follows:\n",
    "\n",
    "- Get the building labels and AOI (most likely from the SpaceNet AWS public dataset bucket)\n",
    "- Use the AOI and the image bounds to determine which images can be used for training and validation\n",
    "- Split the building labels by image, save a label GeoJSON file per image\n",
    "- Split the labeled images into a training and validation set, using the percentage of the AOI each covers, aiming at an 80%/20% split.\n",
    "\n",
    "This process will save the split label files, and `train_scenes.csv` and `val_scenes.csv` files that are used by the experiment at `experiments.spacenet.chip_classification`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from os.path import join\n",
    "import json\n",
    "import tempfile\n",
    "\n",
    "import boto3\n",
    "import botocore\n",
    "import rasterio\n",
    "from shapely.geometry import (Polygon, shape)\n",
    "\n",
    "import rastervision as rv\n",
    "from rastervision.utils.files import (\n",
    "    download_if_needed, list_paths, file_to_json, json_to_file, \n",
    "    get_local_path, make_dir, sync_to_dir, str_to_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the label and AOI data from AWS's public dataset of Space Net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_uri = join(raw_uri, 'AOIs/AOI_1_Rio/srcData/buildingLabels/Rio_Buildings_Public_AOI_v2.geojson')\n",
    "aoi_uri = join(raw_uri, 'AOIs/AOI_1_Rio/srcData/buildingLabels/Rio_OUTLINE_Public_AOI.geojson')\n",
    "\n",
    "label_json = file_to_json(label_uri)\n",
    "aoi_json = file_to_json(aoi_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the AOI to determine what images are inside the training set\n",
    "\n",
    "Here we compare the AOI to the image extents to deteremine which images we can use for training and validation. We're using `rasterio`'s ability to read the metadata from raster data on S3 without downloading the whole image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"100.0\" height=\"100.0\" viewBox=\"-43.7856105430399 -23.02091800403995 0.31122581507989366 0.20602391507999585\" preserveAspectRatio=\"xMinYMin meet\"><g transform=\"matrix(1,0,0,-1,0,-45.8358120929999)\"><path fill-rule=\"evenodd\" fill=\"#66cc99\" stroke=\"#555555\" stroke-width=\"0.006224516301597874\" opacity=\"0.6\" d=\"M -43.596484619999956,-22.95362395099994 L -43.59650719599995,-22.95109540999988 L -43.59647891599991,-22.95037898599992 L -43.59637522299988,-22.94957772099997 L -43.596292080999945,-22.949084805999973 L -43.59576155699989,-22.948589650999963 L -43.594048344999976,-22.947675937999918 L -43.5926777759999,-22.946933546999958 L -43.59119299199989,-22.945505869999977 L -43.59119299199989,-22.94373555099986 L -43.59104681399998,-22.942907205999973 L -43.591821169999946,-22.94207944599998 L -43.593477274999884,-22.941508375999945 L -43.59399123799989,-22.93990937799998 L -43.593705702999955,-22.93836748699988 L -43.592563561999896,-22.937739308999937 L -43.587443556999915,-22.93625720199998 L -43.58426263599989,-22.93533640999982 L -43.58382618099995,-22.938907407999977 L -43.58097082799992,-22.93916698599986 L -43.58017132899994,-22.940194912999914 L -43.57954315099994,-22.941279946999884 L -43.57897208099996,-22.94236498099997 L -43.578229688999954,-22.94350712299996 L -43.577236024999934,-22.94419240799982 L -43.5765735839999,-22.94419240799982 L -43.57520301399984,-22.94333580199998 L -43.57291873199995,-22.94173680399996 L -43.56817723499984,-22.94168792199997 L -43.56659287699995,-22.937666091999915 L -43.564559444999816,-22.937718907999965 L -43.56197144099991,-22.937349193999978 L -43.56162278499994,-22.937271136999925 L -43.56098335499996,-22.935797668999953 L -43.55995542799997,-22.934883955999965 L -43.55652900399997,-22.934883955999965 L -43.556354161999934,-22.936938351999913 L -43.555976778999934,-22.937085111999977 L -43.55489404199989,-22.937639683999976 L -43.55270642799991,-22.93805244099991 L -43.54884249199995,-22.934188504999952 L -43.545651629999895,-22.931588542999975 L -43.543997109999964,-22.929106761999947 L -43.54379066999991,-22.92869388199989 L -43.54391789299996,-22.92715872499997 L -43.5452351859999,-22.926571155999966 L -43.54659707099995,-22.926329530999965 L -43.55563784499998,-22.92603408099984 L -43.55296679299994,-22.91672586699997 L -43.55569693499996,-22.91628422599996 L -43.556352880999896,-22.916571201999886 L -43.556795432999934,-22.91867859199982 L -43.55729718899994,-22.923352844999954 L -43.55758767899994,-22.92604648199989 L -43.563027768999916,-22.926284154999905 L -43.56303233199992,-22.926243343999943 L -43.56424254699988,-22.92794787199989 L -43.56551014099995,-22.93029820299995 L -43.56717385799993,-22.930166161999978 L -43.56976186199995,-22.93003412099995 L -43.57005657899998,-22.92991725099995 L -43.57012048599995,-22.929972748999944 L -43.573147159999905,-22.931743067999946 L -43.57451772999997,-22.931857281999896 L -43.57680201199997,-22.931628852999836 L -43.57817258199998,-22.930486711999833 L -43.57908629499997,-22.92774557299998 L -43.57897208099996,-22.925232861999973 L -43.57600251299982,-22.923234114999957 L -43.57047885999992,-22.922819840999978 L -43.57018439299998,-22.92258700699989 L -43.566940651999914,-22.921002853999937 L -43.566510410999854,-22.919120546999977 L -43.56804675199987,-22.91870691699995 L -43.568874011999924,-22.918056926999895 L -43.57029217299993,-22.915397874999883 L -43.57123761299994,-22.913920624999946 L -43.57194669399996,-22.912797913999952 L -43.570552470999985,-22.91207291799992 L -43.57051274,-22.911069714999883 L -43.57043540699988,-22.90988593399993 L -43.57026193599984,-22.90784509399998 L -43.57060791799995,-22.907803904999923 L -43.571893959999954,-22.90771451699993 L -43.574063412999976,-22.907792305999976 L -43.57419211499996,-22.91143884299987 L -43.57330576399988,-22.915693325999825 L -43.576555715999916,-22.917406935999963 L -43.578623866999976,-22.917288755999948 L -43.57988680899996,-22.917288755999948 L -43.579969907999896,-22.917779794999888 L -43.58001154799996,-22.91798204999992 L -43.58006369199995,-22.91877842199989 L -43.58005749899996,-22.918795143999944 L -43.58014515699983,-22.919814576999954 L -43.58151927799992,-22.921188697999924 L -43.58601012099996,-22.922725038999943 L -43.5889055319999,-22.92319775899989 L -43.59183355299996,-22.92145220799989 L -43.59471066799989,-22.922972968999943 L -43.59586470699992,-22.922794509999846 L -43.596994950999886,-22.921842724999976 L -43.59733997299992,-22.920617302999915 L -43.59711566499993,-22.92024345699997 L -43.598624881999854,-22.919118241999968 L -43.599065081999925,-22.919023063999816 L -43.599374411999975,-22.918178354999895 L -43.59944579599994,-22.917892819999906 L -43.60115900799991,-22.91428793599988 L -43.59825606499987,-22.91390722299991 L -43.59742808499993,-22.913833498999963 L -43.5972923839999,-22.912646107999933 L -43.59541083999994,-22.912704385999973 L -43.594829640999876,-22.908767586999943 L -43.59228294299993,-22.9085695099999 L -43.58556126799988,-22.907117296999957 L -43.58506859299996,-22.906369326999823 L -43.58520472199996,-22.90500803899988 L -43.584383807999984,-22.900261013999966 L -43.58436001299998,-22.899463893999894 L -43.58545456499991,-22.899214050999944 L -43.58532240499994,-22.898589963999882 L -43.58717789799988,-22.89816826099991 L -43.587170695999816,-22.897642554999948 L -43.587279547999856,-22.89739700499996 L -43.58754646099993,-22.896794899999975 L -43.58760797699995,-22.8966561299999 L -43.58775074499994,-22.896406286999877 L -43.587738847999844,-22.895906599999932 L -43.58810766399989,-22.894788253999934 L -43.588500274999944,-22.89466928 L -43.588678734999974,-22.894419436999897 L -43.588976757999944,-22.89349314799989 L -43.59013221599997,-22.89330057199993 L -43.59026107599988,-22.893979236999883 L -43.5904157409999,-22.89438374499997 L -43.59267622899989,-22.89619213599991 L -43.59419908399997,-22.89748894199994 L -43.59396923899993,-22.895575814999916 L -43.594336390999956,-22.894916004999857 L -43.595523614999934,-22.89548453499998 L -43.596587235999834,-22.89560271499994 L -43.59721517899993,-22.89576951199996 L -43.5978753519999,-22.90052275499994 L -43.60173007799983,-22.901474538999935 L -43.60182260499994,-22.901388741999938 L -43.60375261999991,-22.90283083199995 L -43.60596551899988,-22.90103433899992 L -43.60801369799998,-22.901306909999903 L -43.6080160599999,-22.90131028499991 L -43.608344979999856,-22.903544669999974 L -43.60876138599991,-22.9061739739999 L -43.60936814799993,-22.906733147999887 L -43.61111359499995,-22.907611807999956 L -43.61162863599992,-22.91195606499997 L -43.6119974529999,-22.912122626999974 L -43.614243700999964,-22.912675441999966 L -43.61397240499997,-22.91371686599996 L -43.61400128499997,-22.91484894499996 L -43.61311579899996,-22.91485900699996 L -43.61325856699989,-22.91635806799991 L -43.612449549999894,-22.91640565699987 L -43.61186658199989,-22.916703088999952 L -43.61148586899992,-22.916857753999977 L -43.61118843599991,-22.917000521999967 L -43.61104566799992,-22.917214673999922 L -43.61106946299992,-22.91871373399988 L -43.613996199999974,-22.918915987999924 L -43.61586407699997,-22.919142036999972 L -43.61594717799994,-22.91910549199997 L -43.61820784599996,-22.919308598999976 L -43.62006306599994,-22.919470626999896 L -43.62093232899986,-22.919546544999946 L -43.620949073999896,-22.91985034399994 L -43.62101560999997,-22.92105750299993 L -43.62126545299998,-22.922758817999977 L -43.62138442599996,-22.923520244999963 L -43.62193170199993,-22.92360352599991 L -43.62226482699998,-22.923270401999957 L -43.622574156999974,-22.9231871209999 L -43.62293107599993,-22.923258503999932 L -43.62316595199991,-22.923282801999903 L -43.62325230299996,-22.923448860999883 L -43.62515587199988,-22.92426977499997 L -43.62567935299995,-22.9243292619999 L -43.626227556999936,-22.92249488999994 L -43.626500266999926,-22.92239000099994 L -43.62666682999998,-22.92239000099994 L -43.6277375869999,-22.92262794699991 L -43.628903522999906,-22.922937276999846 L -43.62939131299987,-22.923068147999913 L -43.62981961599996,-22.92267553699986 L -43.630592939999985,-22.923008660999983 L -43.63080709199994,-22.922937276999846 L -43.631294880999974,-22.922651741999914 L -43.63168749199997,-22.922330514999828 L -43.632151486999874,-22.92224723399994 L -43.632579789999966,-22.921985492999966 L -43.63340070399994,-22.922128259999965 L -43.63440007799994,-22.922021184999892 L -43.633832060999964,-22.918568004999884 L -43.63457853799997,-22.91766677099986 L -43.63547083499992,-22.917619181999896 L -43.635875343999885,-22.917155186999878 L -43.63726120299998,-22.91724481999995 L -43.640729443999874,-22.919225317999917 L -43.642133325999964,-22.91966551799993 L -43.64217384099993,-22.919671305999884 L -43.642704396999875,-22.92004623199989 L -43.64400120299996,-22.920664891999934 L -43.64522662599995,-22.92105750299993 L -43.646210629999985,-22.921340895999833 L -43.647070707999944,-22.921949800999926 L -43.64856976799996,-22.923377477999964 L -43.649830882999936,-22.92340127199998 L -43.650449541999876,-22.922473281999828 L -43.6500450339999,-22.921521497999947 L -43.64936688799992,-22.921140783999874 L -43.64930207399988,-22.920725976999847 L -43.64949775799994,-22.919356187999938 L -43.64917653099991,-22.917631078999932 L -43.64859356299996,-22.917107597999973 L -43.64673758299989,-22.91621529999992 L -43.6453099069999,-22.915727509999897 L -43.64458417099996,-22.915727509999897 L -43.64427484099997,-22.915905969999983 L -43.644036894999886,-22.916369964999944 L -43.643894126999896,-22.916857753999977 L -43.64414397099989,-22.918071278999946 L -43.643822743999976,-22.91829732799988 L -43.643300987999964,-22.918354040999873 L -43.64187158599998,-22.91796420399993 L -43.642085736999945,-22.915085055999953 L -43.64174071499997,-22.915001774999894 L -43.64120533599993,-22.914930390999928 L -43.64124102799997,-22.914728136999884 L -43.6414670769999,-22.914276038999958 L -43.64174071499997,-22.913835838999944 L -43.64144328299989,-22.91338374099996 L -43.640503857999875,-22.912914028999978 L -43.63706390599992,-22.90907524199997 L -43.622527430999924,-22.90218668899996 L -43.622409578999964,-22.90151171799988 L -43.61945507699994,-22.897493595999947 L -43.61685511599984,-22.89442091499984 L -43.61502332499998,-22.892175493999957 L -43.613841523999895,-22.889220991999935 L -43.61345562999992,-22.88836478999997 L -43.614314244999946,-22.88632558099988 L -43.61596876599987,-22.885557409999933 L -43.61839145699986,-22.88508468999987 L -43.61998688799997,-22.885734680999917 L -43.62376864899994,-22.886443760999896 L -43.62492186899982,-22.886379692999924 L -43.63991881599998,-22.89650421399989 L -43.64011666999983,-22.896348635999914 L -43.64955563399991,-22.904856122999945 L -43.65162576499995,-22.905284425999923 L -43.65469526899983,-22.90578411199988 L -43.65662263299987,-22.908496697999965 L -43.66026320799989,-22.909496071999968 L -43.66526007599998,-22.91006714199989 L -43.66821189399997,-22.911051081999858 L -43.66940033799989,-22.912779727999975 L -43.66954310599988,-22.91534954599996 L -43.66818681299986,-22.92006087799996 L -43.666341460999945,-22.922247962999904 L -43.662560998999936,-22.92216395199989 L -43.66106193899992,-22.922092568999858 L -43.65972943999998,-22.92230671999988 L -43.6582065849999,-22.922639844999935 L -43.654708777999986,-22.923758190999934 L -43.6524958789999,-22.924495823999905 L -43.65152029999996,-22.925518991999922 L -43.651139585999886,-22.92713702599997 L -43.65173445099998,-22.927303587999972 L -43.650996818999886,-22.929920994999918 L -43.650958309999965,-22.93012453999995 L -43.650842153999974,-22.930123248999962 L -43.648819611999954,-22.93002807099998 L -43.64729675699988,-22.930147043999966 L -43.645494274999976,-22.935454350999976 L -43.645467047999944,-22.935534518999873 L -43.64810416199998,-22.93758560899994 L -43.64998537899993,-22.937870641999893 L -43.651815352999904,-22.939610159999916 L -43.65241497699998,-22.939705338999886 L -43.65306219099989,-22.939391249999858 L -43.65311929799998,-22.939267517999895 L -43.65425192099991,-22.93911523299994 L -43.65449938499995,-22.939191374999837 L -43.65505141999989,-22.939400767999928 L -43.655641525999954,-22.93972437499997 L -43.656307774999846,-22.93998135599992 L -43.656802702999926,-22.94030496299996 L -43.657069202999935,-22.94041917699991 L -43.65723100599985,-22.93983858899992 L -43.65803050499983,-22.93998135599992 L -43.658281908999925,-22.93930570799995 L -43.65909650399993,-22.939733891999936 L -43.6598103419999,-22.940266891999897 L -43.66073654899998,-22.940872255999977 L -43.66065767599997,-22.941700423999976 L -43.65889662899997,-22.943141280999896 L -43.65736425599994,-22.944445224999868 L -43.65693595299996,-22.945282795999958 L -43.65691520799993,-22.945499928999936 L -43.656001232999984,-22.94606967899989 L -43.65468022399989,-22.94598711599997 L -43.65365229699995,-22.94595856199993 L -43.65287183399994,-22.945587366999973 L -43.65206281699989,-22.9452066529999 L -43.65122524699996,-22.94533038499992 L -43.650587550999944,-22.94557784899996 L -43.649797569999976,-22.94572061599996 L -43.64835085799996,-22.94615843699995 L -43.64726582399993,-22.946834203999913 L -43.64679944899996,-22.947757434999914 L -43.646694752999906,-22.948909093999816 L -43.647227751999935,-22.94981328899985 L -43.647341965999885,-22.952211785999964 L -43.64726582399993,-22.953277784999955 L -43.64714209199997,-22.954419925999957 L -43.64716206599985,-22.954418058999977 L -43.64687559199996,-22.955714352999962 L -43.647303894999936,-22.957570331999932 L -43.6479796619999,-22.95863633099998 L -43.64905517799997,-22.958988490999957 L -43.6500450339999,-22.959350168999947 L -43.651063442999885,-22.959807025999908 L -43.65141197899993,-22.960083449999956 L -43.651092551999966,-22.960738990999857 L -43.649654802999976,-22.962348289999966 L -43.64932167799992,-22.962462503999973 L -43.64887433899986,-22.962414914999954 L -43.64848410799988,-22.962586235999936 L -43.64812242999989,-22.963252484999884 L -43.64791303699997,-22.963623680999945 L -43.64769412699991,-22.96419475199997 L -43.647551358999976,-22.96485148299996 L -43.64803676899993,-22.96535592899994 L -43.649093249999964,-22.96568905299995 L -43.64967383799984,-22.96607928499992 L -43.650587550999944,-22.966136391999953 L -43.651891495999905,-22.96588892799997 L -43.65269099499989,-22.96558435699984 L -43.65409626899998,-22.965420811999877 L -43.657251575999965,-22.967721266999888 L -43.657992752999974,-22.97141519699983 L -43.65813316899988,-22.975013363999892 L -43.65804954099997,-22.97577796799993 L -43.65801146999996,-22.976567948999843 L -43.65801146999996,-22.977034322999884 L -43.65780207699993,-22.977186608999943 L -43.65769738099988,-22.977548286999934 L -43.65710727399983,-22.978490552999915 L -43.65684077499998,-22.979290051999897 L -43.65674336399991,-22.97987451399996 L -43.65550827699997,-22.98119362099993 L -43.655089490999956,-22.981707584999924 L -43.65444227799992,-22.982278654999902 L -43.65352856499993,-22.982830689999957 L -43.65319544099998,-22.983163814999955 L -43.65306219099989,-22.983554045999938 L -43.65304315499998,-22.98390620699996 L -43.65302411899995,-22.98421077799992 L -43.65346193999994,-22.98501027599997 L -43.65419507099983,-22.985812927999973 L -43.65369869699998,-22.986090789999935 L -43.65379149499984,-22.98633349499994 L -43.65346312999992,-22.98646555499994 L -43.653106210999965,-22.986718966999945 L -43.65333674999994,-22.98693215399993 L -43.653438145999985,-22.987025917999972 L -43.65354522099989,-22.98718653199984 L -43.65305267299988,-22.98746492799995 L -43.65290335699996,-22.987246316999972 L -43.65240664899983,-22.98651909299997 L -43.65187840899989,-22.98672253699982 L -43.65176062599994,-22.986954533999892 L -43.651614288999895,-22.98700807199998 L -43.6514929359999,-22.986890288999916 L -43.65124666199995,-22.986747520999927 L -43.65067650299994,-22.986119667999958 L -43.65032343099995,-22.985134007999932 L -43.649990306999825,-22.98461647599993 L -43.64968097699989,-22.984485604999918 L -43.64936569799994,-22.984455861999948 L -43.64899688199995,-22.984664064999947 L -43.648158416999934,-22.985129123999968 L -43.64778735399989,-22.985334936999948 L -43.647622946999945,-22.985426125999936 L -43.646504395999955,-22.98480088399998 L -43.64588750099989,-22.984396253999876 L -43.64536528999997,-22.984120302999884 L -43.6444580599999,-22.98364089699993 L -43.643928629999834,-22.984438015999956 L -43.6434948939999,-22.98495759599996 L -43.642145223999876,-22.98421077799992 L -43.64184101099994,-22.984078511999883 L -43.64188229299998,-22.983438641999953 L -43.64178711499994,-22.98259988199993 L -43.64171003499996,-22.982439298999964 L -43.64217056499996,-22.98155113499996 L -43.64234188599988,-22.981137108999974 L -43.642463238999824,-22.9806802519999 L -43.642227671999876,-22.98023053399993 L -43.641992104999986,-22.979916444999958 L -43.64008742399989,-22.979545119999955 L -43.640336,-22.978274616999897 L -43.640393106999966,-22.974973812999906 L -43.640393106999966,-22.974148859999957 L -43.640393106999966,-22.971821518999946 L -43.64037883099991,-22.970450948999883 L -43.64046449099993,-22.970208243999878 L -43.64063581299996,-22.969708557999923 L -43.64049304499997,-22.969123209999964 L -43.63983631399998,-22.968580692999865 L -43.63867989599987,-22.968166666999878 L -43.637537753999936,-22.967981068999904 L -43.6357103279999,-22.96799534599984 L -43.6337972419999,-22.968166666999878 L -43.633326107999835,-22.96893761199982 L -43.63322617099993,-22.970251074999965 L -43.63354025999996,-22.971193340999946 L -43.63298811199991,-22.972355420999975 L -43.631259866999926,-22.97311512899995 L -43.62978546999989,-22.97302076699998 L -43.62950972599998,-22.97298525899987 L -43.62693011699997,-22.971650197999907 L -43.62604495799991,-22.970879251999975 L -43.625659484999915,-22.97049378 L -43.62544533299996,-22.969822771999873 L -43.62548816399993,-22.969351637999978 L -43.625745145999986,-22.96870918399992 L -43.62595929699995,-22.968266603999894 L -43.626173448999964,-22.967638426999883 L -43.62641615399991,-22.966838927999902 L -43.62604495799991,-22.965996597999947 L -43.6240319339999,-22.96558257199996 L -43.62297545299998,-22.966225026999894 L -43.62216167699995,-22.96740999799988 L -43.621719097999915,-22.968323710999982 L -43.621547775999886,-22.968837674999975 L -43.62119962799994,-22.969533971999965 L -43.61862103899995,-22.970836421999877 L -43.618310300999894,-22.971118585999875 L -43.61765973699988,-22.970967291999955 L -43.61614283099988,-22.970699602999957 L -43.614447464999955,-22.970699602999957 L -43.61391208599997,-22.970283196999958 L -43.61316850399987,-22.969985764999933 L -43.61096750299993,-22.968439114999967 L -43.61007780799997,-22.967976473999954 L -43.609785522999914,-22.96800226399995 L -43.60949547999991,-22.967825714999947 L -43.60914238299995,-22.96767438799992 L -43.60891539199997,-22.967447396999887 L -43.6085496849999,-22.96582062999994 L -43.60814614599991,-22.963563332999968 L -43.60798221899995,-22.96246573699989 L -43.60781423999998,-22.961840481999957 L -43.59753398099991,-22.96334016299994 L -43.598272046999966,-22.965631470999938 L -43.59899809799998,-22.969095771999946 L -43.601262321999855,-22.974163322999914 L -43.60263938899993,-22.976648255999976 L -43.6078444599999,-22.979354892999936 L -43.61473464799997,-22.983770236999874 L -43.61936262599994,-22.987425041999927 L -43.617321594999964,-22.9932479819999 L -43.616468867999856,-22.994817 L -43.617287485999896,-22.994987544999958 L -43.617287485999896,-22.99536274499991 L -43.617071461999956,-22.99577205299994 L -43.616866806999894,-22.996010816999956 L -43.61657119499995,-22.996454234999874 L -43.61582079599998,-22.997727639999823 L -43.61527505099991,-22.99859173599998 L -43.61486574199989,-22.999262547999876 L -43.61449054199994,-22.999990207999872 L -43.62095566499994,-23.002536403999954 L -43.62310822099994,-23.00788068099996 L -43.630898913999886,-23.009391121999954 L -43.642902940999875,-23.006370240999843 L -43.651448853999966,-23.00366734799985 L -43.65653665299993,-23.000209233999897 L -43.66450626699998,-22.98890505399993 L -43.67126642099993,-22.983136388999924 L -43.67354338499996,-22.97990206399993 L -43.67403500299997,-22.977987343999928 L -43.67455249499994,-22.977159356999834 L -43.67830431099998,-22.97612437299989 L -43.681745631999945,-22.97620199699992 L -43.68195262899985,-22.97664186499992 L -43.68200437799993,-22.977469851999956 L -43.68180068399988,-22.980248824999933 L -43.681714048999936,-22.980415906999895 L -43.68097046699995,-22.985472261999917 L -43.686026821999974,-22.992015778999928 L -43.69316520499996,-22.99305679299988 L -43.704467644999966,-22.98576969399994 L -43.707888119999836,-22.97922617599994 L -43.7069958219999,-22.976103133999914 L -43.70298048199993,-22.972980090999897 L -43.701247734999924,-22.972368532999837 L -43.70075042699989,-22.970582743999955 L -43.70153969899991,-22.970069717999877 L -43.70386841299995,-22.970147340999972 L -43.70632649899994,-22.96986272099997 L -43.70743910699997,-22.969086482999955 L -43.70762022899987,-22.96794800099991 L -43.70793072399988,-22.96580040899994 L -43.708655212999986,-22.963808064999967 L -43.7090494389999,-22.962664808999932 L -43.734657055999946,-22.943831693999982 L -43.754882474999874,-22.932083104999947 L -43.76648234699991,-22.92747289899995 L -43.76899116599992,-22.92289799399998 L -43.773322232999874,-22.92326503299995 L -43.774083660999906,-22.92140905399998 L -43.772564480999904,-22.91638194999996 L -43.774066878999975,-22.913642281999955 L -43.77376944699995,-22.903678288999913 L -43.76514390099993,-22.888955373999977 L -43.74729794299992,-22.87676063699996 L -43.73659036899994,-22.874976040999968 L -43.728559687999905,-22.879586246999963 L -43.729005836999875,-22.88999638899986 L -43.73435962399992,-22.895201459999953 L -43.735951142999966,-22.896960505999914 L -43.73554851299997,-22.89888888799993 L -43.73553639199997,-22.898946941999952 L -43.735450088999926,-22.89892282799991 L -43.73230032499998,-22.89804274699992 L -43.72982568599997,-22.89756685499998 L -43.72735104599991,-22.897281318999887 L -43.72459087199991,-22.89751926499997 L -43.72221141099993,-22.89732890899984 L -43.720600698999874,-22.89732890899984 L -43.71935605699986,-22.894473554999934 L -43.718832575999954,-22.892998288999877 L -43.71611998999998,-22.890904363999937 L -43.703889560999926,-22.881434108999827 L -43.698559567999894,-22.875961348999965 L -43.693515110999954,-22.872677691999968 L -43.6905645789999,-22.87258251399993 L -43.68923208099994,-22.874724028999935 L -43.68794717199995,-22.87796009599998 L -43.68780104399997,-22.87918757099993 L -43.68558067299983,-22.882114423999894 L -43.68528324099998,-22.889104090999922 L -43.68559833399996,-22.893279078999967 L -43.68417679099991,-22.894434082999965 L -43.68189250899985,-22.896789749999982 L -43.679608225999914,-22.897646355999882 L -43.67796639799997,-22.89671836599996 L -43.67746671099985,-22.894077163999896 L -43.679037154999946,-22.891435962999935 L -43.67910853899997,-22.888651992999826 L -43.67746671099985,-22.886796013999913 L -43.67325506499998,-22.88622494299989 L -43.66954310599988,-22.88615355899998 L -43.66675913699993,-22.882727134999982 L -43.66483177299989,-22.880085933999908 L -43.66147673299997,-22.878515488999938 L -43.657907541999975,-22.87930071099987 L -43.656265713999915,-22.880228700999908 L -43.65363918199995,-22.87982901099997 L -43.655919626999946,-22.876389651999887 L -43.662153814999954,-22.862636366999936 L -43.65549132399997,-22.860875565999947 L -43.649447492999855,-22.86206529599997 L -43.645735533999925,-22.86354056199997 L -43.64447713399994,-22.865003579999893 L -43.63927636199992,-22.86816483399997 L -43.63565463699996,-22.87356867799997 L -43.625423169999976,-22.870843992999937 L -43.61927780699983,-22.872025793999967 L -43.61254154399995,-22.875866645999963 L -43.609409771999935,-22.878052976999925 L -43.60296194799997,-22.88276693199998 L -43.60201364999995,-22.881474876999903 L -43.6046234799999,-22.879116596999893 L -43.60482082999994,-22.87852454699987 L -43.59381000399998,-22.86824403199995 L -43.592346547999966,-22.867572036999945 L -43.590848206999965,-22.866884022999955 L -43.590699665999864,-22.865309480999883 L -43.58965865099998,-22.858617246999927 L -43.58787405599992,-22.854899338999815 L -43.58177668699989,-22.850586565999947 L -43.57734898699994,-22.849701025999934 L -43.57473241699989,-22.848121654999886 L -43.566309125999965,-22.846408442999916 L -43.563096852999934,-22.845194917999947 L -43.56109810599986,-22.843338937999874 L -43.55895659099997,-22.838342069999953 L -43.556458156999895,-22.83191752499988 L -43.553103116999864,-22.826992040999926 L -43.5484631679999,-22.826420970999948 L -43.54475120899991,-22.829204939999954 L -43.54260969399991,-22.83684300999994 L -43.54125340099995,-22.845052149999958 L -43.54025402799988,-22.84619429099996 L -43.538312375999965,-22.84666172599998 L -43.53686435999998,-22.846571224999877 L -43.53314645299986,-22.846125075999907 L -43.52987469399994,-22.845827643999883 L -43.52779266499988,-22.84731480699992 L -43.52794138199994,-22.84909940299997 L -43.52883367899989,-22.851032714999974 L -43.52957726099993,-22.852668593999965 L -43.52987469399994,-22.854007040999875 L -43.53046955899998,-22.85608906899995 L -43.532105437999974,-22.8572788 L -43.53567463,-22.8572788 L -43.53872296099985,-22.857173684999964 L -43.54132584999991,-22.859658260999936 L -43.54385402699995,-22.860699274999888 L -43.546530920999885,-22.861442856999872 L -43.547125785999924,-22.86382231799996 L -43.54697706999997,-22.865606912999965 L -43.54742321899994,-22.869176104999895 L -43.54816679999982,-22.872001714999897 L -43.55203342399989,-22.873786310999947 L -43.55768464399995,-22.87452989199994 L -43.557974938999905,-22.87452989199994 L -43.55825571499997,-22.87523183299993 L -43.56353811799994,-22.875731519999874 L -43.56882052199995,-22.875874287999977 L -43.571818641999926,-22.875160448999964 L -43.57581613699995,-22.87416107599995 L -43.5786001059999,-22.873375853999903 L -43.58001668099996,-22.873131616999956 L -43.5839419699999,-22.873975764999898 L -43.58562591799989,-22.87500077699997 L -43.58596396099989,-22.879817887999934 L -43.58439266099998,-22.879884751999953 L -43.580171306999944,-22.88018627699995 L -43.57399003799992,-22.881693903999917 L -43.57234316899991,-22.88220500099993 L -43.561034537999944,-22.882242949999977 L -43.56048322799995,-22.881775648999906 L -43.55906506699995,-22.88029839799998 L -43.55758781599985,-22.87923477699991 L -43.55339242399998,-22.87870296699998 L -43.54854704199994,-22.87769843699982 L -43.54523799999987,-22.876752995999937 L -43.54257894899996,-22.874921204999964 L -43.527629170999944,-22.874389394999923 L -43.5266246409999,-22.88130292799997 L -43.53271091399995,-22.883548348999966 L -43.54163350799996,-22.887566470999957 L -43.545379387999844,-22.88849192399988 L -43.54609894799995,-22.888779747999934 L -43.5505575819999,-22.890730400999928 L -43.55203335399989,-22.895070904999955 L -43.5523288039999,-22.897966316999884 L -43.55221062399983,-22.89914811699998 L -43.55102882299991,-22.89956174699995 L -43.54842886199998,-22.899502656999914 L -43.546478890999936,-22.899502656999914 L -43.54562314799995,-22.899632314999906 L -43.54228349899995,-22.898379946999853 L -43.5330063639999,-22.895779985999923 L -43.532533643999955,-22.896961785999906 L -43.53466088499994,-22.898498126999925 L -43.53525178499996,-22.900802637999846 L -43.53495633499995,-22.903402598999946 L -43.534069984999974,-22.90541166 L -43.53347908399991,-22.90665255099998 L -43.53353817399994,-22.907597990999875 L -43.534188164999875,-22.909902502999955 L -43.533892713999876,-22.912325193999948 L -43.533774533999974,-22.9132706339999 L -43.532297283999924,-22.91433425499997 L -43.52993368299997,-22.91634331599994 L -43.52793205699987,-22.91667085499995 L -43.527146912999854,-22.915100566999968 L -43.52798371199998,-22.91007977299995 L -43.52544283999998,-22.90434803999989 L -43.52307923899991,-22.900270827999975 L -43.51653947899996,-22.898517413999855 L -43.51448446499995,-22.895520517999955 L -43.51112942499998,-22.891879942999935 L -43.5069891629999,-22.888096599999983 L -43.50420519299996,-22.885669549999875 L -43.49806618399987,-22.88545539799992 L -43.49385453799994,-22.887668296999948 L -43.490356729999974,-22.889238740999872 L -43.48635923599994,-22.899303860999908 L -43.48798194299985,-22.90464308999998 L -43.48591161,-22.90606165099996 L -43.48608888099989,-22.909252511999966 L -43.49744270599996,-22.941724452999892 L -43.499763054999846,-22.943257782999922 L -43.50980826499983,-22.949895845999947 L -43.518931721999934,-22.96517998799993 L -43.524763735999954,-22.979039598999975 L -43.53211627099995,-22.982180487999983 L -43.536898986999915,-22.98539275999991 L -43.54567919899995,-22.984678921999944 L -43.55181820799993,-22.985107224999922 L -43.55481632899995,-22.987605658999883 L -43.55595847,-22.99217422399994 L -43.55795721699997,-22.99617171899996 L -43.562668549999955,-22.99674278899994 L -43.56780818599992,-22.994815425999832 L -43.570592154999986,-22.989533021999932 L -43.57344750899989,-22.98253740699994 L -43.57523210399995,-22.98068142799991 L -43.57701669999989,-22.979039598999975 L -43.57801607399989,-22.975399023999955 L -43.57908683099993,-22.969331398999884 L -43.57594594299991,-22.965262519999953 L -43.569898585999965,-22.95996522299987 L -43.568933667999886,-22.954807899999935 L -43.5690693,-22.954483561999893 L -43.570877690999964,-22.953436598999872 L -43.57292402699994,-22.953151063999883 L -43.575779379999915,-22.953674544999956 L -43.57806366299991,-22.953626955999823 L -43.58134731899992,-22.95310347399993 L -43.583166517999985,-22.952992094999956 L -43.587538139999936,-22.954528785999912 L -43.5920495979999,-22.955499605999933 L -43.59467652299992,-22.955042749999905 L -43.59644684199992,-22.953786393999962 L -43.596484619999956,-22.95362395099994 z\" /></g></svg>"
      ],
      "text/plain": [
       "<shapely.geometry.polygon.Polygon at 0x7f4011303b38>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aoi = shape(aoi_json['features'][0]['geometry'])\n",
    "aoi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_uri = join(raw_uri, 'AOIs/AOI_1_Rio/srcData/mosaic_3band')\n",
    "image_paths = list_paths(images_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['s3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223103.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223112.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223113.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223121.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223123.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223130.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223131.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223132.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223133.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223301.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223310.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223311.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232002.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232003.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232020.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232021.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232022.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232023.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232200.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232201.tif']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image_paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bounds_to_shape(bounds):\n",
    "    return Polygon([[bounds.left, bounds.bottom],\n",
    "                    [bounds.left, bounds.top],\n",
    "                    [bounds.right, bounds.top],\n",
    "                    [bounds.right, bounds.bottom],\n",
    "                    [bounds.left, bounds.bottom]])\n",
    "\n",
    "image_to_extents = {}\n",
    "for img in image_paths:\n",
    "    with rasterio.open(img, 'r') as ds:\n",
    "        image_to_extents[img] = bounds_to_shape(ds.bounds)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "intersecting_images = []\n",
    "for img in image_to_extents:\n",
    "    if image_to_extents[img].intersects(aoi):\n",
    "        intersecting_images.append(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['s3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223123.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223130.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223131.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223132.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223133.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232020.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232021.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232022.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232023.tif',\n",
       " 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232200.tif']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "intersecting_images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Match labels to images\n",
    "\n",
    "Find the labels that intersect with the image's bounding box, which will be saved into a labels geojson that matches the image name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a class_id and class_name to the properties of each feature\n",
    "for feature in label_json['features']:\n",
    "    feature['properties']['class_id'] = 1\n",
    "    feature['properties']['class_name'] = 'building'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_to_features = {}\n",
    "for img in intersecting_images:\n",
    "    image_to_features[img] = []\n",
    "    bbox = image_to_extents[img]\n",
    "    for feature in label_json['features']:\n",
    "        if shape(feature['geometry']).intersects(bbox):\n",
    "            image_to_features[img].append(feature)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processed_labels_uri = join(processed_uri, 'labels')\n",
    "\n",
    "for img in image_to_features:\n",
    "    fc = {}\n",
    "    fc['type'] = 'FeatureCollection'\n",
    "    fc['crs'] = label_json['crs']\n",
    "    fc['features'] = image_to_features[img]\n",
    "    img_id = os.path.splitext(os.path.basename(img))[0]\n",
    "    label_path = join(processed_labels_uri, '{}.geojson'.format(img_id))\n",
    "    json_to_file(fc, label_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split into train and validation\n",
    "\n",
    "Split up training and validation data. There's an odd shaped AOI and not that many images, so we'll split the train and validation roughly based on how much area each scene covers of the AOI. \n",
    "\n",
    "Create a CSV that our experiments will use to load the training and validation data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split training and validation\n",
    "ratio = 0.8\n",
    "aoi_area = aoi.area\n",
    "images_to_area = {}\n",
    "for img in intersecting_images:\n",
    "    area = image_to_extents[img].intersection(aoi).area\n",
    "    images_to_area[img] = area / aoi_area\n",
    "\n",
    "train_imgs = []\n",
    "val_imgs = []\n",
    "train_area_covered = 0\n",
    "for img in sorted(intersecting_images, reverse=True, key=lambda img: images_to_area[img]):\n",
    "    if train_area_covered < ratio:\n",
    "        train_imgs.append(img)\n",
    "        train_area_covered += images_to_area[img]\n",
    "    else:\n",
    "        val_imgs.append(img)\n",
    "print(\"{} training images with {}% area.\".format(len(train_imgs), train_area_covered))\n",
    "print(\"{} validation images with {} area.\".format(len(val_imgs), 1 - train_area_covered))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_split_csv(imgs, path):\n",
    "    csv_rows = []\n",
    "    for img in imgs:\n",
    "        img_id = os.path.splitext(os.path.basename(img))[0]\n",
    "        img_path = join('AOIs', 'AOI_1_Rio', 'srcData/mosaic_3band', '{}.tif'.format(img_id))\n",
    "        label_path = join('labels','{}.geojson'.format(img_id))\n",
    "        csv_rows.append('\"{}\",\"{}\"'.format(img_path, label_path))\n",
    "    str_to_file('\\n'.join(csv_rows), path)\n",
    "        \n",
    "save_split_csv(train_imgs, join(processed_uri, 'train-scenes.csv'))\n",
    "save_split_csv(val_imgs, join(processed_uri, 'val-scenes.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}