Duplicate NDCs within a single SPL

[jrlegrand]

Problem Statement
Individual SPLs sometimes have more than one NDC within the XML (and within the "Ingredients and Appearance" section). This prevents us from:

1. Using NDC as a primary key
2. Using NDC as a lookup_field

Example: https://dailymed.nlm.nih.gov/dailymed/drugInfo.cfm?setid=2524b253-069e-4028-819c-361b888df110

Criteria for Success
Either modify Scrapy logic to remove duplicates (if appropriate) or modify Django Rest Framework to use an auto-incrementing number as the primary key and don't use NDC as a lookup_field.

Success = 0 duplicate errors during Scrapy run.

Additional Information
The two XML documents I found are below:

• 3c8e9c87-0475-444b-bbe7-99620519a581.xml
• 8d4d72be-638b-11ea-918e-832cfc2ca371.xml

I pulled this from the API for one of them:

GET /spl/8d4d72be-638b-11ea-918e-832cfc2ca371/
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
{
    "id": "8d4d72be-638b-11ea-918e-832cfc2ca371",
    "set": "http://192.168.1.12:8000/set/2524b253-069e-4028-819c-361b888df110/",
    "ndcs": [
        "http://192.168.1.12:8000/ndc/50458-178-00/",
        "http://192.168.1.12:8000/ndc/50458-178-15/",
        "http://192.168.1.12:8000/ndc/50458-178-00/",
        "http://192.168.1.12:8000/ndc/50458-178-20/",
        "http://192.168.1.12:8000/ndc/50458-178-00/",
        "http://192.168.1.12:8000/ndc/50458-178-12/",
        "http://192.168.1.12:8000/ndc/50458-178-28/",
        "http://192.168.1.12:8000/ndc/50458-178-06/",
        "http://192.168.1.12:8000/ndc/50458-176-00/",
        "http://192.168.1.12:8000/ndc/50458-176-15/",
        "http://192.168.1.12:8000/ndc/50458-176-28/",
        "http://192.168.1.12:8000/ndc/50458-176-06/"
    ]
}

[jrlegrand]

@yevgenybulochnik NDC List only has one of these products listed (50458-178-15) because it filters out products with expired marketing end dates: https://ndclist.com/ndc/50458-178 ... Also, it doesn't even find a result for the 50458-178-00 NDC which makes me think it doesn't care about internal pack NDCs maybe.

[jrlegrand]

Interestingly, DailyMed returns all the NDCs in its API. https://dailymed.nlm.nih.gov/dailymed/services/v2/spls/2524b253-069e-4028-819c-361b888df110/ndcs.json

[finish06]

Closed by mistake. Reopening issue.

[finish06]

@yevgenybulochnik Please close this if it was completed by the linked refractor above. Thanks.

[jrlegrand]

I think it may be fine to close this for now, but let's think about it. Eventually we may want to do something different about this because multiple different outer pack NDCs can point to the same inner pack NDC. So we will have multiple "NDC"s in our database that are fundamentally the same. My thought is that maybe the scraper will need to change to only add NDCs if they don't already exist -- unless we can think of a good reason to store each of these inner pack NDCs as a different object in our database -- for instance, if we are somehow mapping their relation to the outer pack. Though I'm sure there are smarter ways to do that without having duplicate rows.