S3

S3 ๋…ธ๋“œ๋Š” Amazon S3์— ์ €์žฅ๋œ ๋‹จ์ผ ํŒŒ์ผ์„ Unstructured API์™€ ์—ฐ๋™ํ•˜์—ฌ ํ…์ŠคํŠธ๋กœ ๋ณ€ํ™˜ํ•˜๊ณ , ๋ฌธ์„œ ๊ฐ์ฒด(Document)๋กœ ์ถœ๋ ฅํ•ฉ๋‹ˆ๋‹ค. OCR, ๋ ˆ์ด์•„์›ƒ ๋ถ„์„, Chunking ๋“ฑ ๊ณ ๊ธ‰ ์ „์ฒ˜๋ฆฌ ์˜ต์…˜์„ ํ™œ์šฉํ•˜์—ฌ ์ •๊ตํ•œ ๋ฌธ์„œ ํŒŒ์‹ฑ์ด ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค.


์ฃผ์š” ๊ธฐ๋Šฅ

  • ์ง€์ •ํ•œ S3 ๋ฒ„ํ‚ท์˜ ๊ฐœ๋ณ„ ํŒŒ์ผ์„ ๋ถˆ๋Ÿฌ์™€ ํ…์ŠคํŠธ๋กœ ๋ณ€ํ™˜

  • Unstructured API์™€ ์—ฐ๋™ํ•˜์—ฌ ๋ฌธ์„œ ์œ ํ˜•์— ๋”ฐ๋ฅธ ์ž๋™ ํŒŒ์‹ฑ

  • OCR ๋ฐ ๊ณ ํ•ด์ƒ๋„ ๋ชจ๋ธ์„ ํ†ตํ•œ ๋ ˆ์ด์•„์›ƒ ๊ธฐ๋ฐ˜ ๋ถ„์„ ์ง€์›

  • Chunk ๋‹จ์œ„ ๋ถ„ํ• , ํƒœ๊ทธ ์œ ์ง€, ํŽ˜์ด์ง€ ๊ตฌ๋ถ„ ๋“ฑ ์„ธ๋ฐ€ํ•œ ์ „์ฒ˜๋ฆฌ ์˜ต์…˜ ์ œ๊ณต

WindyFlo S3
WindyFlo S3 Parameters
WindyFlo S3 Parameters

์ž…๋ ฅ๊ฐ’ (Inputs)

ํ•ญ๋ชฉ
์„ค๋ช…
ํ•„์ˆ˜ ์—ฌ๋ถ€

AWS Credential

S3 ์ ‘๊ทผ์šฉ ์ธ์ฆ ์ •๋ณด (Credential์— ๋“ฑ๋ก)

ํ•„์ˆ˜

Bucket

๋Œ€์ƒ S3 ๋ฒ„ํ‚ท ์ด๋ฆ„

ํ•„์ˆ˜

Object Key

๋Œ€์ƒ ํŒŒ์ผ ๊ฒฝ๋กœ ๋ฐ ์ด๋ฆ„ (์˜ˆ: doc/sample.pdf)

ํ•„์ˆ˜

Region

๋ฒ„ํ‚ท ๋ฆฌ์ „ (์˜ˆ: us-east-1)

ํ•„์ˆ˜

Unstructured API URL

๋ฌธ์„œ ๋ถ„์„์„ ์œ„ํ•œ Unstructured API ์„œ๋ฒ„ ์ฃผ์†Œ

ํ•„์ˆ˜

Unstructured API KEY

Unstructured API ์ธ์ฆ ํ‚ค

ํ•„์ˆ˜

Text Splitter

๋ถ„์„๋œ ํ…์ŠคํŠธ๋ฅผ ๋ถ„ํ• ํ•˜๋Š” ๋ฐ ์‚ฌ์šฉํ•  Text Splitter ๋…ธ๋“œ

์„ ํƒ


ํŒŒ๋ผ๋ฏธํ„ฐ (Parameters)

ํ•ญ๋ชฉ
์„ค๋ช…

Strategy

๋ฌธ์„œ ์ถ”์ถœ ์ „๋žต (Auto, Hi-Res, Fast ๋“ฑ)

Encoding

๋ฌธ์„œ ์ธ์ฝ”๋”ฉ ๋ฐฉ์‹ (์˜ˆ: utf-8)

Skip Infer Table Types

ํ‘œ ์ถ”๋ก ์„ ๊ฑด๋„ˆ๋›ธ ํŒŒ์ผ ํƒ€์ž… (jpg, png, pdf ๋“ฑ)

Hi-Res Model Name

๊ณ ํ•ด์ƒ๋„ ๋ชจ๋ธ ์„ ํƒ (์˜ˆ: detectron2_onnx)

Chunking Strategy

ํ…์ŠคํŠธ ๋ถ„ํ•  ๊ธฐ์ค€ (์˜ˆ: By Title, By Element ๋“ฑ)

OCR Languages

OCR ๋ถ„์„์— ์‚ฌ์šฉํ•  ์–ธ์–ด (์˜ˆ: eng, kor ๋“ฑ)

Source ID Key

๋ฌธ์„œ ์ถœ์ฒ˜ ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ ํ‚ค ์ด๋ฆ„

Coordinates

๊ฐ ํ…์ŠคํŠธ ๋ธ”๋ก์˜ ์ขŒํ‘œ ์ •๋ณด ํฌํ•จ ์—ฌ๋ถ€

XML Keep Tags

XML ํƒœ๊ทธ ์œ ์ง€ ์—ฌ๋ถ€

Include Page Breaks

ํŽ˜์ด์ง€ ๊ตฌ๋ถ„ ํƒœ๊ทธ ์‚ฝ์ž… ์—ฌ๋ถ€

Multi-Page Sections

๋ฉ€ํ‹ฐํŽ˜์ด์ง€ ์„น์…˜ ๋ณ‘ํ•ฉ ์ฒ˜๋ฆฌ ์—ฌ๋ถ€

Combine Under N Chars

์ง€์ • ๊ธธ์ด ์ดํ•˜ ํ…์ŠคํŠธ ๋ณ‘ํ•ฉ ๊ธฐ์ค€ (๋ฌธ์ž ์ˆ˜)

New After N Chars

์ง€์ • ๊ธธ์ด ์ดํ›„ ํ…์ŠคํŠธ ๋ถ„ํ•  ๊ธฐ์ค€ (๋ฌธ์ž ์ˆ˜)

Max Characters

Chunk ๋‹น ์ตœ๋Œ€ ๋ฌธ์ž ์ˆ˜ (๊ธฐ๋ณธ๊ฐ’: 500)

Additional Metadata

์‚ฝ์ž…ํ•  ์ปค์Šคํ…€ ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ (JSON)

Omit Metadata Keys

์ œ๊ฑฐํ•  ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ ํ‚ค (์‰ผํ‘œ๋กœ ๊ตฌ๋ถ„)


์ถœ๋ ฅ๊ฐ’ (Outputs)

์ถœ๋ ฅ ํ•ญ๋ชฉ
์„ค๋ช…

Document

Unstructured API ๋ถ„์„ ๊ฒฐ๊ณผ๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ์ƒ์„ฑ๋œ ๋ฌธ์„œ ๊ฐ์ฒด ๋ฆฌ์ŠคํŠธ

Text

์ „์ฒด pageContent๋ฅผ ๋ณ‘ํ•ฉํ•œ ํ…์ŠคํŠธ ๋ฌธ์ž์—ด


ํ™œ์šฉ ์˜ˆ์‹œ

  • ์Šค์บ” ๊ธฐ๋ฐ˜ PDF๋ฅผ OCR ๋ถ„์„ํ•˜์—ฌ ํ…์ŠคํŠธ๋กœ ์ถ”์ถœ ๋ฐ ์š”์•ฝ

  • ์ •ํ˜• ๋ฆฌํฌํŠธ ๋ฌธ์„œ๋ฅผ ์„น์…˜๋ณ„๋กœ ๋ถ„ํ•  ํ›„ ๊ฒ€์ƒ‰ ๊ธฐ๋ฐ˜ AI ์‘๋‹ต์— ํ™œ์šฉ

  • ๋‹ค์–‘ํ•œ ์–ธ์–ด์˜ ๊ณ„์•ฝ์„œ, ์ •์ฑ… ๋ฌธ์„œ ๋“ฑ์„ ๊ตฌ์กฐํ™”ํ•˜์—ฌ ๋น„๊ต ๋ถ„์„


์‚ฌ์šฉ ํŒ

  • Strategy๋ฅผ Auto๋กœ ์„ค์ •ํ•˜๋ฉด ํŒŒ์ผ ์œ ํ˜•์— ๋”ฐ๋ผ ์ตœ์  ์ถ”์ถœ ๋ฐฉ์‹ ์ž๋™ ์ ์šฉ

  • OCR ๊ธฐ๋ฐ˜ ์ถ”์ถœ ์ •ํ™•๋„๋ฅผ ๋†’์ด๋ ค๋ฉด Hi-Res Model๊ณผ OCR Language๋ฅผ ํ•จ๊ป˜ ์„ค์ •

  • Text Splitter์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•˜๋ฉด ๊ณผ๋„ํ•œ ๊ธธ์ด ๋ฌธ์ œ๋ฅผ ๋ฐฉ์ง€ํ•˜๊ณ  ์ฒ˜๋ฆฌ ์•ˆ์ •์„ฑ ํ–ฅ์ƒ


์ฃผ์˜์‚ฌํ•ญ

  • Unstructured API ์„œ๋ฒ„ ์ฃผ์†Œ ๋ฐ ํ‚ค ์ž…๋ ฅ์ด ๋ˆ„๋ฝ๋˜๋ฉด ๋ฌธ์„œ ๋ถ„์„์ด ์‹คํ–‰๋˜์ง€ ์•Š์Œ

  • ์ผ๋ถ€ ์ด๋ฏธ์ง€ ๊ธฐ๋ฐ˜ PDF๋Š” ์ •ํ™•ํ•œ ํ…์ŠคํŠธ ์ถ”์ถœ์„ ์œ„ํ•ด OCR ์˜ต์…˜ ํ•„์ˆ˜

  • ๋ณ‘ํ•ฉ/๋ถ„ํ•  ๊ธฐ์ค€์ด ๊ณผ๋„ํ•˜๊ฒŒ ์„ค์ •๋˜๋ฉด ์‘๋‹ต ๋ˆ„๋ฝ ๋˜๋Š” ๊ณผ๋ถ„ํ•  ๊ฐ€๋Šฅ์„ฑ ์žˆ์Œ

Last updated