[ENG-939] MISO API Pricing Data #493
Conversation
gridstatus/miso_api.py
Outdated
| if verbose: | ||
| logger.info(f"Getting data from {url}") | ||
|
|
||
| response = requests.get(url, headers=headers, verify=False) |
There was a problem hiding this comment.
Unfortunately, this doesn't work unless verify=False. The error is:
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))
I've tried using verify=certifi.where() which fails with the same error. We don't want to skip validation so I'm looking into other solutions.
There was a problem hiding this comment.
Running a test against the server shows an incomplete chain of certificates. It seems that a browser is able to find those missing certificates, but requests does not.
https://www.ssllabs.com/ssltest/analyze.html?d=apim.misoenergy.org
https://stackoverflow.com/a/66111417/5755357
| @@ -11,3 +11,7 @@ PJM_API_KEY= | |||
| ERCOT_API_USERNAME= | |||
| ERCOT_API_PASSWORD= | |||
| ERCOT_API_SUBSCRIPTION_KEY= | |||
|
|
|||
| # Sign up here and subscribe to the pricing product | |||
| # https://data-exchange.misoenergy.org/apis | |||
There was a problem hiding this comment.
The Sign Up link here just redirects to the Sign In page so I don't actually know how to create a new account.
There was a problem hiding this comment.
i remember it was bit confusing. it's fine. we dont need to leave documentation here about how to sign up
| # NOTE: this method does not use the support_date_range decorator. Instead | ||
| # it takes the output of a decorated function and processes that output all at once | ||
| # which is more efficient than processing each iteration of the decorator |
There was a problem hiding this comment.
Hopefully, this is clear. We don't want to call process_pricing_data on each loop of the decorated function, so, we return the raw data lists, then combine them into one list and do all the processing at once.
WillKoehrsen
force-pushed
the
miso-api-lmp
branch
from
2b32e60 to
375cf1a
Compare
gridstatus/decorators.py
Outdated
| @@ -40,10 +40,11 @@ def date_range_maker(start, end, freq, inclusive="neither"): | |||
| # current or latest endpoints that are automatically handled. Currently cannot refactor this confidently | |||
| # without improved testing since it touches many methods | |||
| class support_date_range: | |||
| def __init__(self, frequency, update_dates=None): | |||
| def __init__(self, frequency, update_dates=None, convert_to_dataframe=True): | |||
There was a problem hiding this comment.
maybe call convert_to_dataframe-->return_raw. basically, this option returns a list of exactly what the decorated function return.
There was a problem hiding this comment.
Updated to return_raw=False
| @@ -11,3 +11,7 @@ PJM_API_KEY= | |||
| ERCOT_API_USERNAME= | |||
| ERCOT_API_PASSWORD= | |||
| ERCOT_API_SUBSCRIPTION_KEY= | |||
|
|
|||
| # Sign up here and subscribe to the pricing product | |||
| # https://data-exchange.misoenergy.org/apis | |||
There was a problem hiding this comment.
i remember it was bit confusing. it's fine. we dont need to leave documentation here about how to sign up
|
|
||
| node_to_type_mapping = ( | ||
| MISO() | ||
| ._get_node_to_type_mapping() |
There was a problem hiding this comment.
our approach here is to query the {today}_da_expost_lmp.csv file for to obtain node maps. this feels unreliable for historical days when nodes may have been different. that file seems to have data going back to 20220101. perhaps we can can pass the date of the end of the requested range (but not earlier than 20220101) to this function to get a better file.
There was a problem hiding this comment.
I pulled the data from 2022-01-01 and 2024-12-11 and compared the mapping of Node to Type. There were no differences. (The 2024 data has more nodes but none of the mappings have changed).
| REAL_TIME_15_MIN = "REAL_TIME_15_MIN" | ||
| REAL_TIME_HOURLY = "REAL_TIME_HOURLY" | ||
|
|
||
| REAL_TIME_HOURLY_EX_POST_PRELIM = "REAL_TIME_HOURLY_EX_POST_PRELIM" |
There was a problem hiding this comment.
should we update MISO to correctly use these new market names rather than Markets.REAL_TIME_HOURLY_FINAL, Markets.REAL_TIME_HOURLY_PRELIM, Markets.DAY_AHEAD_HOURLY?
perhaps we also consider making a breaking change downstream in our api to just get the "market" column and the dataset names correct
There was a problem hiding this comment.
should we update MISO to correctly use these new market names
Based on our discussion, it sounds like no.
WillKoehrsen
force-pushed
the
miso-api-lmp
branch
from
1bcf66c to
88acf30
Compare
WillKoehrsen
force-pushed
the
miso-api-lmp
branch
from
88acf30 to
f6402e6
Compare
Summary
Details
get_lmp_day_ahead_hourly_ex_anteget_lmp_day_ahead_hourly_ex_postget_lmp_real_time_hourly_ex_post_prelimget_lmp_real_time_hourly_ex_post_finalget_lmp_real_time_5_min_ex_anteget_lmp_real_time_5_min_ex_post_prelimget_lmp_real_time_5_min_ex_post_final