Helioclim-3 automatic access

Last update: Aug. 2016
    Principle - Build wget requests - Error messages - FAQ

1_ Principle

  • What for? To automatize the retrieve of data. Particularly suited for monitoring applications (large number of repetitive requests).
  • How? (build your first request in 5 minutes)
    1. Once you have subscribed an annual access to the HelioClim-3 data (Contact SoDa support by email), use a login/password authentication (advantage: the requests can be launched from any computer)
    2. Then: let's admit that you need to retrieve the 15 min interval irradiation values for the year 2012 and for a tilted plane (25 deg, facing South):
    3. Double click on the executable. You should retrieve the CSV file named "my_first_wget_request.csv". If you face any problem, check the section "3_ FAQ" and the section "4_ Error messages".
  • When? The data of day "d" are available approx. at midnight the same day (HelioClim-3 version 4) and at midnight the next day (HelioClim-3 version 5).

2_ Build wget requests

Request (Unix and Windows):
wget -O output.csv --header="soda-user: your_user" --header=soda-passwd: your_password" ...
"http://server/pub/hc3v5_request.php?
geopoint=latitude,longitude&elevation=altitude& ...
firstday=yyyy-mm-dd&lastday=yyyy-mm-dd&duration=time_step&time=time_reference& ...
slope=tilt_of_panels&azimuth=az_of_panels&albedo=ground_albedo&horizon=boolean_horizon& ...
soda-user=your_login"

With (inputs):

  • login/password authentication: since Nov. 2014, you can use a login and password to access the data from the computer of your choice: specify your_login and your_password parameters (unused if IP authentication).
  • server: "pro.soda-is.com" (main server) or "pro.soda-is.com" (backup server)
  • latitude and longitude: in degrees, with at least 3 digits after comma, e.g. "geopoint=45.563,7.25"
  • altitude: in meters. Set "elevation=-999" to let SoDa get the elevation from Nasa SRTM database
  • yyyy-mm-dd: set the first and the last days of the request. e.g. "firstday=2010-01-01&lastday=2010-01-31"
  • time_step: "15" (minutes), "h" (hourly), "d" (daily), "m" (monthly)
  • time_reference: "TU" (universal time) or "TST" (True Solar Time)
  • tilt_of_panels: from "0" (horizontal) to "90", and "-999" (2D tracking)
  • az_of_panels: from "0" (North), 90° (East), 180° (South) to 270° (West), and "-999" (2D tracking)
  • ground_albedo: from "0.0" to "1.0". A usual inland value is "0.2"
  • boolean_horizon: takes into account "1" or not "0" the far horizon. If not set, default value is "0".

Example: wget -O Madrid_HChourDNI_15to31Aug2010.csv "http://pro.soda-is.com/pub/hc3v5_request.php? ...
geopoint=40.4,-3.68&elevation=-999&firstday=2010-08-15&lastday=2010-08-31&duration=h& ...
time=TU&slope=-999&azimuth=-999&albedo=0.2&horizon=0"

Outputs: identical to those retrieved from the SoDa website, except that the separator is the space instead of the semi-colon, and that there is no header. The time period max retrieved using the wget command is: one month for a 1 to 5 min time step, one year for a 10 min up to one hour time step, and unlimited for the other time steps.

Warning! It is strongly recommanded to avoid the launch of parallel requests, using the "&" at the end of each wget request line. This would endanger our system.

Top of page

3_ FAQ

I_ "I launched this example script, but I can't retrieve the output file!"

In this case, you probably receive a "time out message".

  1. Network shortage in OUR side: to check this point, try the mirror SoDa server ("http://pro.soda-is.com/" or "http://pro.soda-is.com/").
  2. Network shortage on YOUR side: just try to go on the internet.
  3. Problem of proxy: your internet browser knows which "door" (proxy, firewall...) to use to get out of your local network, but you need to tell wget the information relative to this proxy.
    For wget, you need to specify the proxy information in your .bat file by adding:
    set http_proxy=machine:port
    before the wget command. 1-2-3... launch!
    The proxy information (machine and port) are available:
        Internet Explorer: Tools / Options / Connections / Network parameters
        Firefox: Tools / Options / Advanced / Network / Parameters
    Alternatively, you may ask for the proxy information to your IT person.
  II_ "Why two requests inside the same pixel MSG return 2 different irradiation values?"

The HelioClim databases consist of Global Irradiance values over the horizontal plane. These values take into account the default elevation value from the SRTM database (if "elevation=-999"). Two requests made in the same pixel return different results due to a correction with respect to the elevation.

  III_ "How can I change the spaces by semi colons in my output file?"                              

Solution 1 "manual": in the top menu "Data" in Excel (Microsoft Office Excel 2007), select the sub-menu "convert". Select the "space" character and Excel will automatically display one figure per column.

Solution 2 "automatic":
In Windows: click here to download the "sed" set up program which includes the required dynamic link libraries. To replace automatically the " " by ";" in your test.txt file obtained using a wget command, you can use the following command:
sed "s/ /;/g" test.txt > test.csv
As a consequence, you can create a .bat file with the following command lines: wget -O test.txt "http://pro.soda-is.com/pub/hc3v5_request.php?geopoint=45,5&elevation=-999& ...
firstday=2005-01-01&lastday=2005-05-31&duration=m&time=TU&slope=-999&azimuth=-999&albedo=0.2"
sed "s/ /;/g" test.txt > test.csv
del test.txt

=> the file test.txt will be automatically deleted, and replaced by the test.csv file with ";" as separator.

In Unix, sed is a default tool, so you just have to copy and paste the previous example in a .sh file, by taking care of beginning the file by #!/bin/sh.

  IV_ "How to launch automatically my requests at night?";
Example in Unix:
#!/bin/sh
TIMESTAMP=`date --date='2 days ago' +%Y-%m-%d`
wget -O /mypath/output_file_name_$TIMESTAMP.csv "http://pro.soda-is.com/pub/hc3v5_request.php? ...
geopoint=47.3,-1.6&elevation=-999&slope=20&azimuth=145&albedo=0.2&duration=15&time=TU&firstday= ...
$TIMESTAMP&lastday=$TIMESTAMP"

The tool in Unix to launch automatically a .sh file is "cron".

 

Example in Windows: download and unzip this example of executable.
Use the "scheduled tasks" to launch it automatically in a Windows environment.

   
  V_ "I would like to use "curl" instead of "wget". How can I proceed?"               
The "curl" command is particularly suited for a large number of stations (typically more than 100).

Download the curl executable for Windows with the adequate dll.

Request (Unix): curl -F 'stations=@stations.txt' -o result_curl.txt --header 'soda-user: your_login' ...
--header 'soda-passwd:
your_password'
http://server/pub/hc3v5_request.php

Request (Windows): curl -F "stations=@stations.txt" -o result_curl.txt --header "soda-user: your_login" ...
--header "soda-passwd:
your_password"
http://server/pub/hc3v5_request.php

With (inputs):

  • login/password authentication: since Nov. 2014, you can use a login and password to access the data from the computer of your choice: specify your_login and your_password parameters (unused if IP authentication). With IP identification, just omit the soda-user and the soda-password parameters
  • server: "pro.soda-is.com" (main server) or "pro.soda-is.com" (backup server)
  • stations.txt file format:
    #id lat lon elevation UT time_step first_date last_date slope azimuth albedo [horizon]
    st_Paris 48.86 2.33 -999 UT d 2006-05-01 2006-05-31 0.0 0.0 0.2 1
    st_Madrid 40.4 -3.68 -999 UT h 2010-08-15 2010-08-31 -999 -999 0.2 1
    st_Berlin 52.51 13.4 -999 UT 15 2010-09-07 2010-09-07 30 180 0.2 1
    ...
    if [horizon] is not set, the default value is "0" (the shadowing effect of the far horizon is not taken into account).

  VI_ "How can I introduce a delay between two successive wget requests?"               

In Unix: sleep [N]

In Windows: timeout [N]

...where "N" is the number of seconds to wait.

Top of page

 VI_ "Why doesn't SoDa do the switch to the mirror website by itself? "
This issue has already been raised by several customers. Let's name this company "X". "X" suggested that one solution could be based on DNS or IP Failover. Unfortunately, this type of solution doesn't work when two servers are geographically far away from each other.

To handle this case, a third machine is required which would receive all the requests, and would send them to the two servers depending on their availability. The weak point of this solution is that it adds a risk of failure due to the use of an alternative machine.

Then, "X" has decided that the solution we recommend is the most suited in our case: each customer machine-to-machine of SoDa ("X", you, …) must implement a small code which sends the request on one server, and in case of failure, re-send the same request to the mirror website. Please take a look to the example provided in question V.

Top of page

4_ Error messages

  • ERROR -1: Incorrect tilted plane parameter
  • ERROR -2: Cannot open file
  • ERROR -3: Cannot write to file
  • ERROR -4: Cannot read file
  • ERROR -5: Not enough memory
  • ERROR -6: Cannot access remote data
  • ERROR -7: Helioclim service is corrupted
  • ERROR -8: Unregistered IP address
  • ERROR -9: Incorrect location parameter
  • ERROR -10: Incorrect time step parameter
  • ERROR -11: Incorrect date parameter
  • ERROR -12: Incorrect horizon parameter
  • ERROR -13: Incorrect elevation parameter
  • ERROR -14: Incorrect stations file
  • ERROR -15: Subprogram failed with no error message
  • ERROR -16: Incorrect calibration parameter
  • ERROR -17: Incorrect user/password parameters
  • ERROR -18: Incorrect albedo parameter
  • ERROR -19: Incorrect time reference parameter
  • ERROR -21: Incorrect rely parameter
  • ERROR -22: Incorrect header parameter
  • ERROR -23: HC3 database is not updated
  • ERROR -24: This service is out the scope of your SoDa subscription
  • ERROR -25: Incorrect gamma-sun-min parameter
  • ERROR -26: Mismatch dimensions of returned parameters
  • ERROR 1: No data available with these input parameters
    (generic message if the error is unknown)

Top of page