hadoop - Azure Data Lake Large Lookup -
this more of design query, please move if required.. have high volume of logistics supply chain data being stored , analysed within azure data lake service... it's written out azure data warehouse once ready 'serve up'.
we have couple of situations need 'lookup' history of items in supply chain , see has happened them @ some unknown, prior time in past.. lookup table has propensity very large.
a question might be.. given item x being packed item y, has item y ever been marked defective? don't know @ point in time have been defective..
maybe yesterday, maybe last week, maybe 6 months ago - , struggling bit... keep returning, in mind, vast table in database can scan (with suitable indexes) , verify if have ever seen prior event behaviour...
how problem best solved? can't partition data on need scan on anyway don't know when previous event happened or happened - rules out partition on time or physical location.
thanks
Comments
Post a Comment