Actually, this is more of a thing and is actually breaking. For some reason every time I use it I get that same error and nothing is returned. I'm having a hard time troubleshooting this more to give you more info.

When I step through,kernel_api seems to return success, but the raw_text field is equal to NA, so when you try to make ds it says: Error: 'NA' does not exist in current working directory ('*****').. I'm not totally sure what kernel_api is supposed to be returning in the raw_text field though.

ETA: I see now - raw_text seems to be NA for some reason, but if in kernel_api, I changed the line that gets the content of the POSTL

raw_text<- as.character(httr::content(result, type = "text/csv", as = "parsed"))

I can get valid data down from this project, so it's not the project that's the problem (I was suspicious). But clearly that isn't the payload your other functions are looking for so it breaks later. But I can get column types and such. I wonder if some of the other packages have been updated and thus behaving oddly?

@vortexing, thanks for digging in. I see what you're saying about a package dependency possibly changed. May I aska few questions?

1. What version of REDCap?

2. Does this problem persist with the GitHub version? (i.e., install with remotes::install_github(repo="OuhscBbmc/REDCapR"))

3. usually I ask to try with a subset of columns (or rows), but that's tricky with metadata. (i.e., you'd probably have to copy the project and delete columns until it starts working.)

   If you're willing to duplicate the project and fill it with ~2 rows of fake data, I'm happy to look at the problem. Either attach the file in this issue, or email it to [email protected]

REDCap version is 9.3.6
When using the current master branch for REDCapR, I don't get the error but the data frame I get back is totally formatted incorrectly. hmm.... I can't really duplicate the project either b/c I don't have those privileges.

The data frame that comes back has the column names all messed up (they are some of my field names!), and then the entire data frame is duplicated field names that are valid, but all jumbled into one data frame.

I will continue to dig a bit.

With the current github version I can again, change the parsing/text/csv parameters in the httr call in kernal_api and it can at least do that call without returning an NA for the content.

IF I do the following, the redcap_metadata_read will work for me:

• in kernel_api, change

raw_text <- httr::content(result,"parsed", "text/csv")
#raw_text <- gsub("\r\n", "\n", raw_text) # Convert all line-endings to linux-style

• Then kernel$raw_text is actually a perfectly formatted data frame that I can assign to ds and I'm off and running (if I do not do your assignment to ds):

    try (
      ds <- readr::read_csv(kernel$raw_text, col_types = col_types), #Convert the raw text to a dataset.
      silent = TRUE #Don't print the warning in the try block.  Print it below, where it's under the control of the caller.
    )

ETA: I'm not sure what possible REDCap metadata structure you're anticipating might cause an error if the above changes were in there, but for my, very simple REDCap project, the metadata structure that comes down is pretty straightforward and always parsable into a csv.

@vortexing thanks for narrowing it down to httr::content(). I don't see any changelog/release entries in REDCap or httr that are possible suspects.

I assume you're using relatively recent version of httr & readr (and curl, which is underneath httr)? I don't any other REDCapR dependencies that are likely involved here.

I started a new branch, will you see if this fixes it? remotes::install_github("OuhscBbmc/REDCapR", ref = "metadata-content").

I'd still like to see your codebook, because I'm hoping there's something in there that is making httr:content() guess the wrong type. In the new branch, I'm forcing it to be "text/csv", like your suggestion above.

However I'm reluctant to ask httr to parse it into a tibble (which it does using readr::read_csv()). I think there are some caller of kernel_api() that need the plain text (not a tibble). Also, the httr::content() documentation advises against it in this scenario.

When using content() in a package, DO NOT use on as = "parsed". Instead, check the mime-type is what you expect, and then parse yourself. This is safer, as you will fail informatively if the API changes, and you will protect yourself against changes to httr.

If this doesn't fix it, do you have time for a webex? If so, can you email me your address at [email protected]? This metadata function is called by important functions like redcap_read(), so I'd like to make sure it's solid for all scenarios and REDCap installations.

@vortexing, even if it does work, will you please tell me the value of result? (Using browser(), or however you did your debugging above.)

I'm particularly curious about Content-Type and charset.

Response [https://bbmc.ouhsc.edu/redcap/api/]
  Date: 2019-11-11 23:23
  Status: 200
  Content-Type: text/csv; charset=utf-8
  Size: 1.56 kB
field_name,form_name,section_header,field_type,field_label,select_choices_...
record_id,demographics,,text,"Study ID",,,,,,,,,,,,,
name_first,demographics,"Contact Information",text,"First Name",,,,,,y,,,,,,,
...

If there's nothing suspicious, will you see if REDCapR::read_metadata() works for another project on the same server? I guess there's a small chance the encoding is set at the server-level.

This just in! I took a gander at my data dictionary when manually downloading it from a browser. I noticed two strange characters in one of my field descriptions. I deleted that, reuploaded it, then tested your functions again and it worked!!!

So I think now I can't do the above testing b/c content type now is correct like you have above.

I think the lesson here is that there CAN be things in a REDCap data dictionary that won't break REDCap but will impact whether or not we can parse it normally in R when it gets pulled down. I have no idea where those characters would've come from, why they were in only one field description and I don't remember what they were (one was the copyright symbol but that's allI remember).

I think your function is fine, BUT there must be things that break httrbut not REDCap.

Sorry for the sort of false alarm - thanks for helping me figure out where the problem was!!!

That's great news. I'm glad it worked out.

I'd like to make REDCapR routines as robust to this as possible (using nondynamic defaults and helpers like redcap_column_sanitize()). I'd like to include your old data dictionary in the package's test suite to make ensure that this specific problem doesn't occur again (and hopefully it will generalize to other bad characters/encodings too).

The metadata is used by several other important REDCapR functions, and I don't want problems to cascade to other areas. If REDCapR can't gracefully fix it, I'd at least like to give a better error message so the user can jump on it quicker.

Is it ok with you and your team if you send me the bad data dictionary (or at least record_id and the problematic columns)? Is the project is production? If so, can you download the old data dictionary with the 'Product Revision History' tab? If you don't have REDCap privileges, maybe Bas or @plitwin can find who does?

Judging from the great debugging skills that you demonstrated above, if it took you some time to isolate and identify the problem, most REDCap users don't have a chance if it happens to them. And this may not be such a fluke --I'm wondering if a bad dictionary was uploaded after modifying it in Excel (and Excel added the copyright symbol and other problem).

This project is not in production right now, but I actually had never had to look up a previous version of the data dictionary, so thanks for that tidbit!

I will email you the previous data dictionary and the current. It's a small project but you can see what changes I made between the broken vs fixed versions. There were small formatting changes, removal of extra lines in field descriptions, etc, I made too that I don't THINK were the issue. Perhaps in this test setup you can have both the formatting marks and weird symbols (turns out it WASN'T a copyright, it was a ?鿡 on line 127 in the csv I'll send you.

Thanks for working on this package!! It's saved me a TON of time and let me build on REDCap for really cool other things too!

Note to others: @vortexing worked out some things and sent me the problematic lines in the data dictionary. It's now in the test suite as problematic-dictionary.csv and test-metadata-read.R.

If anyone encounters something similar, please consider

1. creating a new issue that references this one (ie, include the text "Issue redcap_metadata_read, "object 'ds' not found"  #269")
2. in the issue, attach the your data dictionary, I'll isolate the problematic fields and include them in the live version of problematic-dictionary.csv. If you're uncomfortable sharing your dictionary with the world, either (a) isolate the fields yourself before attaching it, or (b) email it to me.

If we can't find a way for REDCapR to gracefully adapt, we'll at least try to provide a better error message.

Thanks again to @vortexing and @plitwin!