Skip to main content

Processing Uploads in the Background

Uploaded data shouldn't be sent to Quirrel, but be stored in your own database. Add a new DB entity for it:

model UploadedCSV {  id          Number @id @default(autoincrement())  data        String}

Then when a user uploads something, you insert it into the database and enqueue the resulting record's ID into Quirrel.

// app/mutations/uploadCsvForProcessingimport db from "db"import csvProcessingQueue from "app/api/process-csv"
export default async function uploadCsvForProcessing(data: string) {  const record = await db.uploadedCsv.create({    data: {data},  })
  await csvProcessingQueue.enqueue(record.id)
  return record.id}

Our Quirrel Queue then fetches the corresponding data from the database and does the required processing. After that's done, it deletes the database record (alternative: add a flag called "finishedProcessing" and set it to true).

// app/api/process-csvimport db from "db"import {Queue} from "quirrel/blitz"
export default Queue("api/process-csv", async (uploadId: number) => {  const upload = await db.uploadedCsv.findUnique({    where: {id: uploadId},  })
  await doYourProcessing(upload.data)
  await db.uploadedCsv.delete({where: {id: uploadId}})})

Now when you want to know whether an upload has already been processed, you can look it up in your own database:

// app/queries/hasFinishedProcessingimport db from "db"
export default async function hasFinishedProcessing(uploadId: number) {  const count = await db.uploadedCsv.count({where: {uploadId}})  return count === 0}