3.4 KiB
We're building a service that can be used to extract and store facts from different types of materials. Users will be able to create new tables to store different types of facts in teh DB. The flow will be like this. There will be some folder with a set of data in it, could be text files, html pages, pdf or a mix of a bunch of stuff. That directory will need to be onboarded to the service. They will onboard with info like the folder directory and the name of the table the facts about this directory should be stored in, and the prompt they'd like to give the AI to influence the types of facts that come out. By default, all fact retrieval will have the same prompt Extract key facts from the following article in structured JSON format. Return only valid JSON without any additional text.
Article Title: {title}
Article Content: {article_content[:3000]}...
Extract the following information:
1. Main topic/subject
2. Key entities (companies, people, locations, organizations)
3. Financial impact or implications
4. Key dates or time periods mentioned
5. Summary of main points
Format the response as a JSON object with these fields:
{{
"title": "{title}",
"summary": "brief summary",
"main_topic": "main topic",
"key_entities": ["entity1", "entity2"],
"financial_impact": "positive/negative/neutral",
"key_dates": ["date1", "date2"],
"main_points": ["point1", "point2", "point3"]
}}. That's the prmoopt you'll send to the ai. They can also pick waht ai they'd like to use to extract facts. By default it will be gpt-oss, but they can also choose qwen3 and qwen3-coder. Once the directory has been onboarded, the onboarded file should become reachable behind some ismple ftp server that you set up. Be careful that users of the ftp server can only ever navigate into directories that are actually onboarded. You should not allow the ftp server to server every file on a machine, only the file underneath the scope onboarded. For instance, an onboarded dir of dir1/file1 should only every show file1. it shoudl not show a dir2/file2 if that dir or a parent of that dir was never onboarded. Next you'll need to set up a Fact Extraction job that runs a from a cron job every 10 minutes or so. the Job of the FactExtractor will be to pull extract facts from all the files that have not yet been sent through for fact extraction. You'll need to keep a cache of all the files you've already extracted facts from. You'll also need to use data transformer when necessary to turn a file into text. For instance, if you get a website, use extractors like newspaper3k or beautiful soup, to extract mainly text from the pages and not a lot of html tags. If you get a pdf, use a library to turn it to plain text. This goes for any unique file types. You can't process media just yet, but in teh future we'll add vision models that you can use. Dont' worry abotu that for now. After extraction jobs, add the facts into a DB. The user can choose the type of DB during the fact onboarding, but by default, we will use Sqlite. To query for facts out of the sqlite database, expose a REST API for people to get their data using the table name and sending a query. You should give an endpoint that also tells all the tables available and current record count.