25 Commits

Author SHA1 Message Date
kris
6b3774b543 fix: restore visible auth form for douyin workbench 2026-03-21 05:05:44 +08:00
kris
7171dae91c feat: add dedicated douyin workbench entry 2026-03-21 04:57:18 +08:00
kris
9f921fff94 chore: add business smoke check script 2026-03-21 04:52:20 +08:00
kris
39216d18b4 chore: add reliable local business run scripts 2026-03-21 02:59:52 +08:00
kris
c09a976628 feat: upgrade douyin work list filters and ranking 2026-03-21 02:36:18 +08:00
kris
1fb39e040f fix: compact token auth ui 2026-03-21 02:28:44 +08:00
kris
be94836e3c fix: collapse duplicate douyin analysis history 2026-03-21 02:26:42 +08:00
kris
c4222755b1 feat: deepen douyin commercial workbench 2026-03-21 01:34:46 +08:00
kris
f6462dbccc feat: add douyin workbench results ui 2026-03-21 00:52:23 +08:00
kris
741fe4f983 fix: harden douyin control panel auth inputs 2026-03-20 23:31:27 +08:00
kris
5d9c9cf048 feat: add douyin browser control panel 2026-03-20 22:27:54 +08:00
kris
5c52476a45 perf: streamline douyin browser sync handling 2026-03-20 19:41:31 +08:00
kris
4356c46b9e fix: guard douyin creator-center identity merges 2026-03-20 19:31:29 +08:00
kris
10820595cf fix: harden douyin browser capture persistence 2026-03-20 15:14:12 +08:00
kris
1fa1b586f7 feat: add browser-assisted douyin capture flow 2026-03-20 14:51:22 +08:00
kris
7070c3aa85 feat: restore android build path and update status docs 2026-03-20 14:17:33 +08:00
kris
ac6a8a82df feat: add account sync entry and cleanup legacy runtime 2026-03-20 14:10:30 +08:00
kris
98722a580a docs: record huobao override path and credential blocker 2026-03-20 13:47:20 +08:00
kris
e1010503ae docs: record huobao upstream smoke status 2026-03-20 13:26:45 +08:00
kris
1a055a16c2 docs: record douyin workbench validation 2026-03-20 13:19:35 +08:00
kris
f96a37a236 feat: harden douyin sync diagnostics and manual fallback 2026-03-20 13:18:45 +08:00
kris
a906e0ceda feat: formalize live collector douyin deployment 2026-03-20 13:13:03 +08:00
kris
1c539abc6e feat: add content source sync pipeline and harden asr timeouts 2026-03-20 10:11:04 +08:00
kris
63af810236 feat: auto stage real-cut inputs to cutvideo 2026-03-20 06:57:53 +08:00
kris
b145363111 feat: migrate orchestration to n8n and validate lan mvp 2026-03-18 10:05:00 +08:00
46 changed files with 11422 additions and 319 deletions

View File

@@ -2,15 +2,34 @@ DEFAULT_EXTERNAL_BASE_URL=http://test.hyzq.net:8081
LOCAL_OPENAI_BASE_URL=http://127.0.0.1:8317/v1
LOCAL_OPENAI_MODEL=GLM-5
LOCAL_OPENAI_API_KEY=
FASTGPT_BASE_URL=http://127.0.0.1:3000
FASTGPT_DATASET_API_KEY=
# Host-side collector runs can keep using N8N_BASE_URL.
N8N_BASE_URL=http://127.0.0.1:5670
# Dockerized collector should use the internal n8n service address.
COLLECTOR_N8N_BASE_URL=http://n8n:5678
N8N_ANALYSIS_WEBHOOK_PATH=/webhook/storyforge-analysis
N8N_REAL_CUT_WEBHOOK_PATH=/webhook/storyforge-real-cut
N8N_AI_VIDEO_WEBHOOK_PATH=/webhook/storyforge-ai-video
N8N_CONTENT_SOURCE_SYNC_WEBHOOK_PATH=/webhook/storyforge-content-source-sync
ORCHESTRATOR_SHARED_SECRET=storyforge-local-secret
CUTVIDEO_BASE_URL=
CUTVIDEO_API_KEY=
CUTVIDEO_BASE_CONFIG=example.job.yaml
CUTVIDEO_POLL_INTERVAL_SEC=10
CUTVIDEO_MAX_WAIT_SEC=1800
CUTVIDEO_UPLOAD_TIMEOUT_SEC=1800
HUOBAO_BASE_URL=http://127.0.0.1:5678
HUOBAO_POLL_INTERVAL_SEC=10
HUOBAO_MAX_WAIT_SEC=900
YTDLP_BIN=yt-dlp
FFMPEG_BIN=ffmpeg
WHISPER_BIN=
WHISPER_MODEL=./data/collector/models/ggml-base.en.bin
POSTGRES_DB=fastgpt
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
ASR_HTTP_BASE_URL=
ASR_HTTP_TRANSCRIBE_PATH=/transcribe
ASR_HTTP_FIELD_NAME=wav
ASR_HTTP_TIMEOUT_SEC=120
N8N_IMAGE=docker.n8n.io/n8nio/n8n:latest
WEBHOOK_URL=http://127.0.0.1:5670/
GENERIC_TIMEZONE=Asia/Shanghai
TZ=Asia/Shanghai
CLIPROXY_IMAGE=storyforge/cli-proxy-api:patched

4
.gitignore vendored
View File

@@ -20,9 +20,13 @@ build/
.kotlin/
**/.gradle/
**/.kotlin/
node_modules/
**/node_modules/
# Runtime data and artifacts
data/
!android-app/app/src/main/java/com/aiglasses/app/data/
!android-app/app/src/main/java/com/aiglasses/app/data/**
output/
*.log

119
README.md
View File

@@ -5,36 +5,143 @@ StoryForge 现在拆成独立项目目录,和 `AI-glasses` 分开维护。
## 目录
- `android-app/`StoryForge Android 客户端
- `collector-service/`FastAPI 后端,提供登录、审批、素材导入、知识库、智能体和 OTA
- `docker-compose.yml`:本地 FastGPT / collector / 基础依赖编排
- `collector-service/`FastAPI 后端,负责用户体系、项目、Agent、任务、内容分析和对外能力接入
- `n8n/`:工作流导出文件,作为流程编排中枢
- `docker-compose.yml`:本地 `collector + n8n + cli-proxy-api` 编排
- `Common/`:项目约束和架构说明
- `data/collector/`SQLite、任务文件、下载产物
- `docs/`:审计、实施计划、联调说明、当前 MVP 状态
## Android
```bash
cd /Users/kris/code/StoryForge/android-app
cd /Users/kris/code/StoryForge-gitea/android-app
./gradlew assembleDebug
```
## Douyin Browser Capture
```bash
cd /Users/kris/code/StoryForge-gitea
./scripts/start_douyin_workbench.sh
```
业务页:
```text
http://127.0.0.1:3618/workbench
```
完整采集控制台:
```text
http://127.0.0.1:3618
```
常用脚本:
```bash
./scripts/start_douyin_workbench.sh
./scripts/status_douyin_workbench.sh
./scripts/stop_douyin_workbench.sh
./scripts/cleanup_debug_ui.sh
```
如果第一次使用,还需要先安装浏览器依赖:
```bash
cd /Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture
npm install
npx playwright install chromium
```
当前本地页面已经拆成两个入口:
- `/workbench`:业务优先的 `Douyin Workbench`,可直接查看账号列表、商业化账号分析、快照详情、相似账号和对标关系
- `/`:完整浏览器辅助采集控制台,同时保留工作台能力
- 作品工作台支持按 `高分作品 / 最新作品 / 全部作品` 切换,并可按综合分、受欢迎程度、商业价值、发布时间、播放、点赞、分享、评论排序
- 作品列表支持 `视频 / 图文` 类型筛选,并可直接打开原作品链接
- 高分作品支持自动化分析,每条作品卡片下都会展示商业判断、复刻计划、运营动作和风险提醒
或者继续用命令行:
```bash
cd /Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture
npm run capture -- \
--profile-url https://www.douyin.com/user/your_account \
--storyforge-username kris \
--storyforge-password 'Asd123456.'
```
说明:
- 这是“真实浏览器 + 人工登录/过挑战 + 自动提取 + 回写 StoryForge”的辅助采集工具
- 默认输出到 `output/playwright/douyin/`
- 本地控制台模式会把每次运行保存到 `output/playwright/douyin/control-panel/`
- 控制台支持“开始采集 -> 浏览器登录 -> 网页点继续 -> 自动同步”的点击式流程
- 详细说明见 `scripts/douyin-browser-capture/README.md`
## Collector Service
```bash
cd /Users/kris/code/StoryForge/collector-service
cd /Users/kris/code/StoryForge-gitea/collector-service
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --host 0.0.0.0 --port 8081 --reload
```
## Docker Compose
```bash
cd /Users/kris/code/StoryForge-gitea
cp .env.example .env
docker compose up -d --build
```
如果 `collector` 跑在 Docker 里,建议保留:
```bash
COLLECTOR_N8N_BASE_URL=http://n8n:5678
```
如果你单独在宿主机启动 `collector-service`,它读取的仍然是:
```bash
N8N_BASE_URL=http://127.0.0.1:5670
```
默认会启动:
- `collector-service``http://127.0.0.1:8081`
- `n8n``http://127.0.0.1:5670`
- `cli-proxy-api``http://127.0.0.1:8317`
默认会创建最高权限账号:
- `kris`
- `Asd123456.`
## 当前架构
- `collector-service` 负责:
- 用户账号、多项目、多 Agent、多任务、多内容源数据边界
- 调用下载器、本地 ASR、本机 OpenAI 兼容模型
- 调用 Windows `cutvideo``huobao-drama`
- 持久化任务、分镜、分析结果、事件日志
- `n8n` 负责:
- 触发 `analysis_pipeline`
- 触发 `content_source_sync_pipeline`
- 触发 `real_cut_pipeline`
- 触发 `ai_video_pipeline`
- FastGPT 已从主流程设计中移除,不再作为运行时依赖
## 说明
- 新注册账号默认 `pending`
- 主管理员审批后才可使用核心业务接口
- 素材入口支持文字、视频链接、视频上传
- 可选对接本机 OpenAI 兼容模型服务和 FastGPT 数据集 API
- 支持 `user -> project -> knowledge base / assistant(agent) / job / content source` 的多租户边界
- 素材入口支持文字、视频链接、视频上传;内容源账号通过 `content_sources` 建模持久化,并可派生父子分析任务
- `cutvideo` 继续运行在 Windows 机器,本系统通过 API 调度
- `huobao-drama` 继续作为 AI 生成视频主链的核心引擎
- 详细审计、阶段计划和联调步骤见 `docs/`

View File

@@ -0,0 +1,50 @@
package com.aiglasses.app.data
import kotlinx.serialization.json.Json
import kotlinx.serialization.ExperimentalSerializationApi
import java.util.concurrent.TimeUnit
import okhttp3.MediaType.Companion.toMediaType
import okhttp3.Protocol
import okhttp3.OkHttpClient
import okhttp3.Request
import okhttp3.logging.HttpLoggingInterceptor
import retrofit2.Retrofit
import retrofit2.create
import com.jakewharton.retrofit2.converter.kotlinx.serialization.asConverterFactory
object ApiClient {
@OptIn(ExperimentalSerializationApi::class)
val json = Json {
ignoreUnknownKeys = true
explicitNulls = false
}
inline fun <reified T : Any> createService(baseUrl: String): T {
val logging = HttpLoggingInterceptor().apply {
level = HttpLoggingInterceptor.Level.BODY
}
val client = OkHttpClient.Builder()
.protocols(listOf(Protocol.HTTP_1_1))
.connectTimeout(12, TimeUnit.SECONDS)
.readTimeout(20, TimeUnit.SECONDS)
.writeTimeout(20, TimeUnit.SECONDS)
.callTimeout(25, TimeUnit.SECONDS)
.addInterceptor { chain ->
val request: Request = chain.request().newBuilder()
.header("Connection", "close")
.build()
chain.proceed(request)
}
.addInterceptor(logging)
.build()
val normalizedBaseUrl = if (baseUrl.endsWith("/")) baseUrl else "$baseUrl/"
return Retrofit.Builder()
.baseUrl(normalizedBaseUrl)
.client(client)
.addConverterFactory(json.asConverterFactory("application/json".toMediaType()))
.build()
.create<T>()
}
}

View File

@@ -0,0 +1,154 @@
package com.aiglasses.app.data
import retrofit2.http.Body
import retrofit2.http.GET
import retrofit2.http.Header
import retrofit2.http.POST
import retrofit2.http.Path
import retrofit2.http.Query
interface ApiService {
@GET("/healthz")
suspend fun healthz(): ApiEnvelope<HealthzData>
@POST("/api/v1/devices/bind-confirm")
suspend fun bindConfirm(
@Body request: BindConfirmRequest
): ApiEnvelope<BindConfirmData>
@POST("/api/v1/ai/sessions")
suspend fun createSession(
@Header("Idempotency-Key") idempotencyKey: String?,
@Body request: CreateSessionRequest
): ApiEnvelope<SessionData>
@POST("/api/v1/ai/sessions/{sessionId}/stop")
suspend fun stopSession(
@Path("sessionId") sessionId: String,
@Body request: StopSessionRequest
): ApiEnvelope<StopSessionData>
@POST("/api/v1/ai/sessions/{sessionId}/heartbeat")
suspend fun heartbeat(
@Path("sessionId") sessionId: String,
@Body request: HeartbeatRequest
): ApiEnvelope<HeartbeatData>
@GET("/api/v1/devices/{deviceId}/status")
suspend fun getDeviceStatus(
@Path("deviceId") deviceId: String
): ApiEnvelope<DeviceStatusData>
@POST("/api/v1/events")
suspend fun postEvent(
@Body request: ClientEventRequest
): ApiEnvelope<EventSavedData>
@POST("/api/v1/events/batch")
suspend fun postEventsBatch(
@Body request: ClientEventBatchRequest
): ApiEnvelope<EventsBatchSavedData>
@POST("/api/v1/ai/sessions/{sessionId}/messages")
suspend fun sendMessage(
@Path("sessionId") sessionId: String,
@Body request: SessionMessageRequest
): ApiEnvelope<ProviderActionData>
@POST("/api/v1/ai/sessions/{sessionId}/scene-role")
suspend fun switchRole(
@Path("sessionId") sessionId: String,
@Body request: SwitchRoleRequest
): ApiEnvelope<ProviderActionData>
@POST("/api/v1/ai/sessions/{sessionId}/interrupt")
suspend fun interruptSession(
@Path("sessionId") sessionId: String,
@Body request: SessionInterruptRequest
): ApiEnvelope<ProviderActionData>
@GET("/api/v1/baidu/activation/query")
suspend fun activationQuery(
@Query("deviceId") deviceId: String,
@Query("appId") appId: String? = null
): ApiEnvelope<ActivationQueryData>
@POST("/api/v1/licenses/reload")
suspend fun reloadLicenses(): ApiEnvelope<ReloadLicensesData>
@GET("/api/v1/admin/overview")
suspend fun adminOverview(): ApiEnvelope<AdminOverviewData>
@GET("/api/v1/app/update/latest")
suspend fun appUpdateLatest(
@Query("platform") platform: String = "android",
@Query("channel") channel: String = "stable",
@Query("currentVersionCode") currentVersionCode: Int
): ApiEnvelope<AppUpdateLatestData>
@GET("/v2/douyin/accounts")
suspend fun listDouyinAccounts(): ApiEnvelope<List<DouyinAccountSummary>>
@POST("/v2/douyin/accounts/sync")
suspend fun syncDouyinAccount(
@Body request: DouyinAccountSyncRequest
): ApiEnvelope<DouyinAccountWorkspace>
@GET("/v2/douyin/accounts/{accountId}")
suspend fun getDouyinAccount(
@Path("accountId") accountId: String
): ApiEnvelope<DouyinAccountWorkspace>
@GET("/v2/douyin/accounts/{accountId}/workspace")
suspend fun getDouyinWorkspace(
@Path("accountId") accountId: String
): ApiEnvelope<DouyinAccountWorkspace>
@GET("/v2/douyin/accounts/{accountId}/snapshots")
suspend fun listDouyinSnapshots(
@Path("accountId") accountId: String
): ApiEnvelope<List<DouyinSnapshotSummary>>
@GET("/v2/douyin/accounts/{accountId}/snapshots/{snapshotId}")
suspend fun getDouyinSnapshot(
@Path("accountId") accountId: String,
@Path("snapshotId") snapshotId: String
): ApiEnvelope<DouyinSnapshotDetail>
@GET("/v2/douyin/accounts/{accountId}/creator-fields")
suspend fun getDouyinCreatorFields(
@Path("accountId") accountId: String
): ApiEnvelope<DouyinSnapshotDetail>
@POST("/v2/douyin/accounts/{accountId}/analysis")
suspend fun analyzeDouyinAccount(
@Path("accountId") accountId: String,
@Body request: DouyinAccountAnalysisRequest
): ApiEnvelope<DouyinAnalysisResult>
@GET("/v2/douyin/accounts/{accountId}/analysis-reports")
suspend fun listDouyinAnalysisReports(
@Path("accountId") accountId: String
): ApiEnvelope<List<DouyinAnalysisReport>>
@POST("/v2/douyin/similar-searches")
suspend fun createDouyinSimilarSearch(
@Body request: DouyinSimilarSearchRequest
): ApiEnvelope<DouyinSimilaritySearchResult>
@GET("/v2/douyin/similar-searches/{searchId}")
suspend fun getDouyinSimilarSearch(
@Path("searchId") searchId: String
): ApiEnvelope<DouyinSimilaritySearchDetail>
@GET("/v2/douyin/accounts/{accountId}/benchmark-links")
suspend fun listDouyinBenchmarkLinks(
@Path("accountId") accountId: String
): ApiEnvelope<List<DouyinLinkedAccount>>
@POST("/v2/douyin/accounts/{accountId}/benchmark-links")
suspend fun createDouyinBenchmarkLinks(
@Path("accountId") accountId: String,
@Body request: DouyinBenchmarkLinkRequest
): ApiEnvelope<DouyinBenchmarkLinkResult>
}

View File

@@ -0,0 +1,276 @@
package com.aiglasses.app.data
import java.util.UUID
class BackendRepository(private var baseUrl: String) {
private var api: ApiService = ApiClient.createService(baseUrl)
fun updateBaseUrl(url: String) {
if (url != baseUrl) {
baseUrl = url
api = ApiClient.createService(baseUrl)
}
}
suspend fun bindDevice(deviceId: String, userId: String): BindConfirmData {
val resp = api.bindConfirm(BindConfirmRequest(deviceId = deviceId, appUserId = userId))
return resp.data
}
suspend fun healthz(): HealthzData {
val resp = api.healthz()
return resp.data
}
suspend fun createSession(deviceId: String, userId: String): SessionData {
val idempotencyKey = "app-${UUID.randomUUID()}"
val resp = api.createSession(
idempotencyKey = idempotencyKey,
request = CreateSessionRequest(deviceId = deviceId, appUserId = userId)
)
return resp.data
}
suspend fun stopSession(sessionId: String): StopSessionData {
val resp = api.stopSession(sessionId, StopSessionRequest())
return resp.data
}
suspend fun heartbeat(sessionId: String): HeartbeatData {
val resp = api.heartbeat(sessionId, HeartbeatRequest())
return resp.data
}
suspend fun getDeviceStatus(deviceId: String): DeviceStatusData {
val resp = api.getDeviceStatus(deviceId)
return resp.data
}
suspend fun postDemoEvent(deviceId: String, sessionId: String?): EventSavedData {
return postEvent(
deviceId = deviceId,
sessionId = sessionId,
eventType = "APP_DEBUG_PING",
eventLevel = "INFO",
payload = mapOf("source" to "android")
)
}
suspend fun postEvent(
deviceId: String,
sessionId: String?,
eventType: String,
eventLevel: String = "INFO",
payload: Map<String, String> = emptyMap()
): EventSavedData {
val resp = api.postEvent(
ClientEventRequest(
sessionId = sessionId,
deviceId = deviceId,
eventType = eventType,
eventLevel = eventLevel,
payload = payload
)
)
return resp.data
}
suspend fun postEventsBatch(events: List<ClientEventRequest>): EventsBatchSavedData {
val resp = api.postEventsBatch(ClientEventBatchRequest(events = events))
return resp.data
}
suspend fun sendMessage(sessionId: String, message: String): ProviderActionData {
val resp = api.sendMessage(
sessionId = sessionId,
request = SessionMessageRequest(message = message)
)
return resp.data
}
suspend fun sendVoiceMessage(
sessionId: String,
pcmBase64: String,
sampleRate: Int,
durationMs: Int,
rms: Int
): ProviderActionData {
val resp = api.sendMessage(
sessionId = sessionId,
request = SessionMessageRequest(
message = "voice_chunk",
messageType = "voice",
extra = mapOf(
"audio_base64" to pcmBase64,
"audio_format" to "pcm_s16le",
"sample_rate" to sampleRate.toString(),
"channels" to "1",
"duration_ms" to durationMs.toString(),
"rms" to rms.toString(),
"encoding" to "base64"
)
)
)
return resp.data
}
suspend fun sendVisionMessage(
sessionId: String,
message: String,
imageBase64: String,
width: Int,
height: Int,
bytes: Int
): ProviderActionData {
val resp = api.sendMessage(
sessionId = sessionId,
request = SessionMessageRequest(
message = message,
messageType = "text",
extra = mapOf(
"image_base64" to imageBase64,
"imageBase64" to imageBase64,
"image" to imageBase64,
"resource_base64" to imageBase64,
"resourceBase64" to imageBase64,
"image_encoding" to "base64",
"imageEncoding" to "base64",
"encoding" to "base64",
"image_format" to "jpeg",
"imageFormat" to "jpeg",
"mime_type" to "image/jpeg",
"mimeType" to "image/jpeg",
"image_width" to width.toString(),
"imageWidth" to width.toString(),
"image_height" to height.toString(),
"imageHeight" to height.toString(),
"image_bytes" to bytes.toString(),
"imageBytes" to bytes.toString(),
"resource_type" to "image",
"resourceType" to "image",
"camera_source" to "android_phone",
"multimodal" to "true",
"with_vision" to "1"
)
)
)
return resp.data
}
suspend fun switchRole(sessionId: String, sceneId: String, roleId: String): ProviderActionData {
val resp = api.switchRole(
sessionId = sessionId,
request = SwitchRoleRequest(sceneId = sceneId, roleId = roleId)
)
return resp.data
}
suspend fun interrupt(
sessionId: String,
interrupt: Boolean,
extra: Map<String, String> = emptyMap()
): ProviderActionData {
val resp = api.interruptSession(
sessionId = sessionId,
request = SessionInterruptRequest(interrupt = interrupt, extra = extra)
)
return resp.data
}
suspend fun activationQuery(deviceId: String): ActivationQueryData {
val resp = api.activationQuery(deviceId = deviceId)
return resp.data
}
suspend fun reloadLicenses(): ReloadLicensesData {
val resp = api.reloadLicenses()
return resp.data
}
suspend fun adminOverview(): AdminOverviewData {
val resp = api.adminOverview()
return resp.data
}
suspend fun appUpdateLatest(currentVersionCode: Int): AppUpdateLatestData {
val resp = api.appUpdateLatest(
platform = "android",
channel = "stable",
currentVersionCode = currentVersionCode
)
return resp.data
}
suspend fun listDouyinAccounts(): List<DouyinAccountSummary> {
val resp = api.listDouyinAccounts()
return resp.data
}
suspend fun syncDouyinAccount(request: DouyinAccountSyncRequest): DouyinAccountWorkspace {
val resp = api.syncDouyinAccount(request)
return resp.data
}
suspend fun getDouyinAccount(accountId: String): DouyinAccountWorkspace {
val resp = api.getDouyinAccount(accountId)
return resp.data
}
suspend fun getDouyinWorkspace(accountId: String): DouyinAccountWorkspace {
val resp = api.getDouyinWorkspace(accountId)
return resp.data
}
suspend fun listDouyinSnapshots(accountId: String): List<DouyinSnapshotSummary> {
val resp = api.listDouyinSnapshots(accountId)
return resp.data
}
suspend fun getDouyinSnapshot(accountId: String, snapshotId: String): DouyinSnapshotDetail {
val resp = api.getDouyinSnapshot(accountId, snapshotId)
return resp.data
}
suspend fun getDouyinCreatorFields(accountId: String): DouyinSnapshotDetail {
val resp = api.getDouyinCreatorFields(accountId)
return resp.data
}
suspend fun analyzeDouyinAccount(
accountId: String,
request: DouyinAccountAnalysisRequest
): DouyinAnalysisResult {
val resp = api.analyzeDouyinAccount(accountId, request)
return resp.data
}
suspend fun listDouyinAnalysisReports(accountId: String): List<DouyinAnalysisReport> {
val resp = api.listDouyinAnalysisReports(accountId)
return resp.data
}
suspend fun createDouyinSimilarSearch(
request: DouyinSimilarSearchRequest
): DouyinSimilaritySearchResult {
val resp = api.createDouyinSimilarSearch(request)
return resp.data
}
suspend fun getDouyinSimilarSearch(searchId: String): DouyinSimilaritySearchDetail {
val resp = api.getDouyinSimilarSearch(searchId)
return resp.data
}
suspend fun listDouyinBenchmarkLinks(accountId: String): List<DouyinLinkedAccount> {
val resp = api.listDouyinBenchmarkLinks(accountId)
return resp.data
}
suspend fun createDouyinBenchmarkLinks(
accountId: String,
request: DouyinBenchmarkLinkRequest
): DouyinBenchmarkLinkResult {
val resp = api.createDouyinBenchmarkLinks(accountId, request)
return resp.data
}
}

View File

@@ -0,0 +1,540 @@
package com.aiglasses.app.data
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
@Serializable
data class ApiEnvelope<T>(
val code: Int,
val message: String,
val traceId: String,
val data: T
)
@Serializable
data class HealthzData(
val status: String = "",
val env: String = "",
val dbPath: String = ""
)
@Serializable
data class BindConfirmRequest(
val deviceId: String,
val deviceSn: String? = null,
val deviceModel: String? = null,
val deviceFwVer: String? = null,
val appUserId: String
)
@Serializable
data class BindConfirmData(
val bindStatus: String,
val licenseStatus: String,
val licenseKeyMasked: String,
val licenseKey: String = ""
)
@Serializable
data class CreateSessionRequest(
val deviceId: String,
val appUserId: String,
val scene: String = "voice_assistant",
val language: String = "zh-CN",
val clientTs: Long? = null
)
@Serializable
data class SessionData(
val sessionId: String,
val provider: String,
val cid: String,
val token: String,
val tokenExpireAt: Long,
val wsUrl: String,
val heartbeatSec: Int,
val appId: String = "",
val context: String = "",
val realtimeWsUrl: String = ""
)
@Serializable
data class StopSessionRequest(
val reason: String = "user_stop"
)
@Serializable
data class StopSessionData(
val sessionStatus: String
)
@Serializable
data class HeartbeatRequest(
val networkType: String? = "wifi",
val bleRssi: Int? = null
)
@Serializable
data class HeartbeatData(
val sessionStatus: String,
val heartbeatAt: Long
)
@Serializable
data class ClientEventRequest(
val sessionId: String? = null,
val deviceId: String,
val eventType: String,
val eventLevel: String = "INFO",
val payload: Map<String, String> = emptyMap(),
val ts: Long? = null
)
@Serializable
data class ClientEventBatchRequest(
val events: List<ClientEventRequest> = emptyList()
)
@Serializable
data class EventSavedData(
val saved: Boolean
)
@Serializable
data class EventsBatchSavedData(
val saved: Int = 0
)
@Serializable
data class SessionMessageRequest(
val message: String,
val messageType: String = "text",
val messageId: String? = null,
val extra: Map<String, String> = emptyMap()
)
@Serializable
data class ProviderActionData(
val status: String = "UNKNOWN",
val detail: String = "",
val asrText: String = "",
val ttsText: String = "",
val audioBase64: String = "",
val audioUrl: String = ""
)
@Serializable
data class SwitchRoleRequest(
val sceneId: String,
val roleId: String,
val extra: Map<String, String> = emptyMap()
)
@Serializable
data class SessionInterruptRequest(
val interrupt: Boolean = true,
val extra: Map<String, String> = emptyMap()
)
@Serializable
data class DeviceStatusData(
val bindStatus: String,
val licenseStatus: String,
val activeSessionId: String? = null,
val activeSessionStatus: String? = null
)
@Serializable
data class AdminStats(
val totalDevices: Int = 0,
val totalSessions: Int = 0,
val runningSessions: Int = 0,
val totalLicenses: Int = 0,
val usedLicenseQuota: Int = 0
)
@Serializable
data class BaiduInfo(
val mode: String = "-",
val generateConfigured: Boolean = false,
val stopConfigured: Boolean = false,
val activationQueryConfigured: Boolean = false
)
@Serializable
data class AdminOverviewData(
val stats: AdminStats = AdminStats(),
val baidu: BaiduInfo = BaiduInfo()
)
@Serializable
data class ActivationQueryData(
val deviceId: String = "",
val appId: String = "",
val status: String = "UNKNOWN",
val detail: String = "",
val licenseKeyMasked: String = ""
)
@Serializable
data class ReloadLicensesData(
val inserted: Int = 0
)
@Serializable
data class AppUpdateLatestData(
val platform: String = "android",
val channel: String = "stable",
val hasUpdate: Boolean = false,
val latestVersionCode: Int = 0,
val latestVersionName: String = "",
val minSupportedCode: Int = 0,
val downloadUrl: String = "",
val apkSha256: String = "",
val releaseNotes: String = "",
val forceUpdate: Boolean = false,
val publishedAt: Long = 0L
)
@Serializable
data class DouyinManualPageCaptureRequest(
val url: String = "",
val title: String = "",
val payload: JsonObject = JsonObject(emptyMap())
)
@Serializable
data class DouyinAccountSyncRequest(
@SerialName("profile_url")
val profileUrl: String = "",
@SerialName("session_cookie")
val sessionCookie: String = "",
@SerialName("creator_center_urls")
val creatorCenterUrls: List<String> = emptyList(),
@SerialName("manual_profile_payload")
val manualProfilePayload: JsonObject? = null,
@SerialName("manual_creator_pages")
val manualCreatorPages: List<DouyinManualPageCaptureRequest> = emptyList(),
@SerialName("manual_work_payloads")
val manualWorkPayloads: List<JsonObject> = emptyList(),
@SerialName("discovery_note")
val discoveryNote: String = ""
)
@Serializable
data class DouyinProfileStats(
val followers: Double = 0.0,
val following: Double = 0.0,
val likes: Double = 0.0,
val videos: Double = 0.0
)
@Serializable
data class DouyinVideoStats(
val play: Double = 0.0,
val like: Double = 0.0,
val comment: Double = 0.0,
val share: Double = 0.0,
val collect: Double = 0.0
)
@Serializable
data class DouyinVideoSummaryItem(
@SerialName("aweme_id")
val awemeId: String = "",
val title: String = "",
val description: String = "",
val tags: List<String> = emptyList(),
@SerialName("published_at")
val publishedAt: String? = null,
val stats: DouyinVideoStats = DouyinVideoStats()
)
@Serializable
data class DouyinVideoSummary(
val count: Int = 0,
@SerialName("top_tags")
val topTags: List<String> = emptyList(),
@SerialName("avg_play")
val avgPlay: Double = 0.0,
@SerialName("avg_like")
val avgLike: Double = 0.0,
@SerialName("avg_comment")
val avgComment: Double = 0.0,
@SerialName("avg_share")
val avgShare: Double = 0.0,
val videos: List<DouyinVideoSummaryItem> = emptyList()
)
@Serializable
data class DouyinAccountSummary(
val id: String = "",
val nickname: String = "",
val signature: String = "",
@SerialName("profile_url")
val profileUrl: String = "",
@SerialName("avatar_url")
val avatarUrl: String = "",
@SerialName("sec_uid")
val secUid: String = "",
@SerialName("douyin_id")
val douyinId: String = "",
@SerialName("profile_stats")
val profileStats: DouyinProfileStats = DouyinProfileStats(),
val tags: List<String> = emptyList(),
val keywords: List<String> = emptyList(),
@SerialName("sync_status")
val syncStatus: String = "",
@SerialName("video_summary")
val videoSummary: DouyinVideoSummary = DouyinVideoSummary()
)
@Serializable
data class DouyinSnapshotSummary(
val id: String = "",
@SerialName("snapshot_type")
val snapshotType: String = "",
@SerialName("source_url")
val sourceUrl: String = "",
@SerialName("field_count")
val fieldCount: Int = 0,
@SerialName("collected_at")
val collectedAt: String = "",
val summary: JsonObject = JsonObject(emptyMap())
)
@Serializable
data class DouyinModelProfileSummary(
val id: String = "",
val name: String = "",
@SerialName("model_name")
val modelName: String = "",
@SerialName("base_url")
val baseUrl: String = "",
@SerialName("is_default")
val isDefault: Boolean = false
)
@Serializable
data class DouyinAnalysisSuggestion(
val id: String = "",
@SerialName("model_profile_id")
val modelProfileId: String = "",
@SerialName("model_label")
val modelLabel: String = "",
val status: String = "",
@SerialName("suggestion_text")
val suggestionText: String = "",
@SerialName("parsed_json")
val parsedJson: JsonElement = JsonObject(emptyMap())
)
@Serializable
data class DouyinAnalysisReport(
val id: String = "",
@SerialName("focus_text")
val focusText: String = "",
@SerialName("model_profile_ids")
val modelProfileIds: List<String> = emptyList(),
@SerialName("linked_account_ids")
val linkedAccountIds: List<String> = emptyList(),
@SerialName("created_at")
val createdAt: String = "",
val suggestions: List<DouyinAnalysisSuggestion> = emptyList()
)
@Serializable
data class DouyinSimilaritySearchPreview(
val id: String = "",
val keywords: List<String> = emptyList(),
@SerialName("created_at")
val createdAt: String = ""
)
@Serializable
data class DouyinLinkedAccount(
@SerialName("relation_id")
val relationId: String = "",
@SerialName("relation_type")
val relationType: String = "",
val note: String = "",
@SerialName("search_id")
val searchId: String = "",
@SerialName("created_at")
val createdAt: String = "",
@SerialName("target_account_id")
val targetAccountId: String? = null,
@SerialName("target_profile_url")
val targetProfileUrl: String = "",
@SerialName("target_nickname")
val targetNickname: String = "",
@SerialName("target_signature")
val targetSignature: String = "",
@SerialName("target_profile_stats")
val targetProfileStats: DouyinProfileStats = DouyinProfileStats(),
@SerialName("target_tags")
val targetTags: List<String> = emptyList()
)
@Serializable
data class DouyinAccountWorkspace(
val account: DouyinAccountSummary = DouyinAccountSummary(),
@SerialName("latest_public_snapshot")
val latestPublicSnapshot: DouyinSnapshotSummary? = null,
@SerialName("latest_creator_snapshot")
val latestCreatorSnapshot: DouyinSnapshotSummary? = null,
@SerialName("linked_accounts")
val linkedAccounts: List<DouyinLinkedAccount> = emptyList(),
@SerialName("recent_reports")
val recentReports: List<DouyinAnalysisReport> = emptyList(),
@SerialName("recent_similarity_searches")
val recentSimilaritySearches: List<DouyinSimilaritySearchPreview> = emptyList(),
@SerialName("available_model_profiles")
val availableModelProfiles: List<DouyinModelProfileSummary> = emptyList(),
@SerialName("sync_errors")
val syncErrors: List<String> = emptyList()
)
@Serializable
data class DouyinAccountAnalysisRequest(
@SerialName("model_profile_ids")
val modelProfileIds: List<String> = emptyList(),
@SerialName("linked_account_ids")
val linkedAccountIds: List<String> = emptyList(),
@SerialName("include_linked_accounts")
val includeLinkedAccounts: Boolean = true,
@SerialName("include_recent_similar_candidates")
val includeRecentSimilarCandidates: Boolean = true,
@SerialName("max_videos")
val maxVideos: Int = 12,
@SerialName("extra_focus")
val extraFocus: String = "",
val temperature: Double = 0.35
)
@Serializable
data class DouyinAnalysisResult(
@SerialName("report_id")
val reportId: String = "",
@SerialName("created_at")
val createdAt: String = "",
val context: JsonElement = JsonObject(emptyMap()),
val suggestions: List<DouyinAnalysisSuggestion> = emptyList()
)
@Serializable
data class DouyinSimilarSearchRequest(
@SerialName("source_account_id")
val sourceAccountId: String? = null,
@SerialName("profile_url")
val profileUrl: String? = null,
@SerialName("candidate_urls")
val candidateUrls: List<String> = emptyList(),
@SerialName("seed_linked_accounts")
val seedLinkedAccounts: Boolean = true,
@SerialName("search_public_pages")
val searchPublicPages: Boolean = true,
@SerialName("model_profile_id")
val modelProfileId: String? = null,
@SerialName("max_candidates")
val maxCandidates: Int = 10,
@SerialName("extra_requirements")
val extraRequirements: String = ""
)
@Serializable
data class DouyinSimilarCandidate(
val id: String = "",
@SerialName("candidate_account_id")
val candidateAccountId: String? = null,
@SerialName("candidate_profile_url")
val candidateProfileUrl: String = "",
@SerialName("candidate_nickname")
val candidateNickname: String = "",
@SerialName("heuristic_score")
val heuristicScore: Double = 0.0,
@SerialName("agent_score")
val agentScore: Double = 0.0,
@SerialName("rationale_text")
val rationaleText: String = "",
val dimensions: JsonElement = JsonObject(emptyMap()),
@SerialName("rank_index")
val rankIndex: Int = 0
)
@Serializable
data class DouyinSimilaritySearchResult(
@SerialName("search_id")
val searchId: String = "",
@SerialName("source_account")
val sourceAccount: DouyinAccountSummary = DouyinAccountSummary(),
@SerialName("model_profile")
val modelProfile: JsonObject = JsonObject(emptyMap()),
@SerialName("raw_model_output")
val rawModelOutput: String = "",
val candidates: List<DouyinSimilarCandidate> = emptyList()
)
@Serializable
data class DouyinSimilaritySearchDetail(
val id: String = "",
@SerialName("source_account_id")
val sourceAccountId: String? = null,
@SerialName("source_profile_url")
val sourceProfileUrl: String = "",
val keywords: List<String> = emptyList(),
val context: JsonElement = JsonObject(emptyMap()),
@SerialName("created_at")
val createdAt: String = "",
val candidates: List<DouyinSimilarCandidate> = emptyList()
)
@Serializable
data class DouyinBenchmarkLinkRequest(
@SerialName("target_account_ids")
val targetAccountIds: List<String> = emptyList(),
@SerialName("target_profile_urls")
val targetProfileUrls: List<String> = emptyList(),
@SerialName("relation_type")
val relationType: String = "benchmark",
val note: String = "",
@SerialName("search_id")
val searchId: String = ""
)
@Serializable
data class DouyinBenchmarkLinkResult(
val saved: Int = 0,
@SerialName("relation_ids")
val relationIds: List<String> = emptyList(),
val links: List<DouyinLinkedAccount> = emptyList()
)
@Serializable
data class DouyinSnapshotField(
@SerialName("field_path")
val fieldPath: String = "",
@SerialName("field_type")
val fieldType: String = "",
@SerialName("field_value_text")
val fieldValueText: String = ""
)
@Serializable
data class DouyinSnapshotDetail(
val id: String = "",
@SerialName("snapshot_type")
val snapshotType: String = "",
@SerialName("source_url")
val sourceUrl: String = "",
@SerialName("field_count")
val fieldCount: Int = 0,
@SerialName("collected_at")
val collectedAt: String = "",
val summary: JsonObject = JsonObject(emptyMap()),
@SerialName("raw_payload")
val rawPayload: JsonElement = JsonObject(emptyMap()),
val fields: List<DouyinSnapshotField> = emptyList()
)

View File

@@ -57,6 +57,9 @@ interface StoryForgeApiService {
@POST("v2/explore/text")
suspend fun createTextJob(@Body request: ExploreTextRequest): JobDto
@POST("v2/pipelines/content-source-sync")
suspend fun createContentSourceSyncJob(@Body request: ContentSourceSyncRequest): JobDto
@Multipart
@POST("v2/explore/upload-video")
suspend fun uploadVideo(

View File

@@ -1,6 +1,10 @@
package com.aiglasses.app.storyforge
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.buildJsonArray
import kotlinx.serialization.json.buildJsonObject
@Serializable
data class RegisterAccountRequest(
@@ -66,12 +70,22 @@ data class PreferredModelRequest(
)
@Serializable
data class KnowledgeBaseDto(
data class ProjectDto(
val id: String,
val user_id: String,
val name: String,
val description: String = "",
val fastgpt_dataset_id: String? = null,
val created_at: String = "",
val updated_at: String = ""
)
@Serializable
data class KnowledgeBaseDto(
val id: String,
val user_id: String,
val project_id: String = "",
val name: String,
val description: String = "",
val sync_status: String = "pending",
val document_count: Int = 0,
val linked_assistant_count: Int = 0,
@@ -82,6 +96,7 @@ data class KnowledgeBaseDto(
@Serializable
data class KnowledgeBaseCreateRequest(
val name: String,
val project_id: String = "",
val description: String = ""
)
@@ -89,12 +104,13 @@ data class KnowledgeBaseCreateRequest(
data class AssistantDto(
val id: String,
val user_id: String,
val project_id: String = "",
val name: String,
val description: String = "",
val system_prompt: String = "",
val generation_goal: String = "",
val knowledge_base_ids: List<String> = emptyList(),
val fastgpt_app_key: String = "",
val config: JsonObject = buildJsonObject { },
val model_profile_id: String = "",
val created_at: String = "",
val updated_at: String = ""
@@ -107,7 +123,7 @@ data class AssistantCreateRequest(
val system_prompt: String = "",
val generation_goal: String = "",
val knowledge_base_ids: List<String> = emptyList(),
val fastgpt_app_key: String = "",
val project_id: String = "",
val model_profile_id: String = ""
)
@@ -118,7 +134,7 @@ data class AssistantUpdateRequest(
val system_prompt: String? = null,
val generation_goal: String? = null,
val knowledge_base_ids: List<String>? = null,
val fastgpt_app_key: String? = null,
val project_id: String? = null,
val model_profile_id: String? = null
)
@@ -126,6 +142,7 @@ data class AssistantUpdateRequest(
data class ExploreVideoLinkRequest(
val video_url: String,
val title: String? = null,
val project_id: String? = null,
val knowledge_base_id: String? = null,
val assistant_id: String? = null,
val analysis_model_profile_id: String? = null,
@@ -136,28 +153,54 @@ data class ExploreVideoLinkRequest(
data class ExploreTextRequest(
val title: String,
val content: String,
val project_id: String? = null,
val knowledge_base_id: String? = null,
val assistant_id: String? = null,
val analysis_model_profile_id: String? = null
)
@Serializable
data class ContentSourceSyncRequest(
val project_id: String = "",
val knowledge_base_id: String = "",
val assistant_id: String = "",
val content_source_id: String = "",
val platform: String = "",
val handle: String = "",
val source_url: String = "",
val title: String = "",
val analysis_model_profile_id: String = "",
val language: String = "auto",
val max_items: Int = 5,
val skip_existing: Boolean = true,
val auto_trigger_analysis: Boolean = true
)
@Serializable
data class JobDto(
val id: String,
val user_id: String,
val project_id: String = "",
val parent_job_id: String = "",
val assistant_id: String? = null,
val knowledge_base_id: String,
val content_source_id: String = "",
val source_type: String,
val line_type: String = "analysis",
val workflow_key: String = "",
val orchestrator: String = "n8n",
val provider_name: String = "",
val provider_task_id: String = "",
val source_url: String? = null,
val title: String,
val language: String,
val status: String,
val transcript_text: String = "",
val style_summary: String = "",
val fastgpt_collection_id: String = "",
val upload_status: String = "pending",
val error: String = "",
val artifacts: Map<String, String> = emptyMap(),
val artifacts: JsonObject = buildJsonObject { },
val result: JsonObject = buildJsonObject { },
val analysis_model_profile_id: String = "",
val created_at: String = "",
val updated_at: String = ""
@@ -173,7 +216,9 @@ data class KnowledgeDocumentDto(
val transcript_text: String = "",
val style_summary: String = "",
val combined_text: String = "",
val fastgpt_collection_id: String = "",
val analysis: JsonObject = buildJsonObject { },
val storyboards: JsonArray = buildJsonArray { },
val source_artifacts: JsonObject = buildJsonObject { },
val analysis_model_profile_id: String = "",
val created_at: String = "",
val updated_at: String = ""
@@ -200,6 +245,7 @@ data class GenerateCopyResponseDto(
@Serializable
data class DashboardDto(
val account: AccountDto,
val projects: List<ProjectDto> = emptyList(),
val knowledge_bases: List<KnowledgeBaseDto> = emptyList(),
val assistants: List<AssistantDto> = emptyList(),
val recent_jobs: List<JobDto> = emptyList(),

View File

@@ -147,6 +147,32 @@ class StoryForgeRepository(private val context: Context) {
)
)
suspend fun createContentSourceSyncJob(
platform: String,
handle: String,
sourceUrl: String,
title: String,
knowledgeBaseId: String,
assistantId: String,
analysisModelProfileId: String,
maxItems: Int,
skipExisting: Boolean,
autoTriggerAnalysis: Boolean
): JobDto = api().createContentSourceSyncJob(
ContentSourceSyncRequest(
knowledge_base_id = knowledgeBaseId,
assistant_id = assistantId,
platform = platform,
handle = handle,
source_url = sourceUrl,
title = title,
analysis_model_profile_id = analysisModelProfileId,
max_items = maxItems,
skip_existing = skipExisting,
auto_trigger_analysis = autoTriggerAnalysis
)
)
suspend fun uploadVideo(
uri: Uri,
title: String,

View File

@@ -293,6 +293,7 @@ private fun ExploreTab(state: StoryForgeUiState, vm: StoryForgeViewModel, onPick
SectionCard(title = "素材入口", subtitle = "视频链接、上传视频、输入文字都会转成文本并做风格分析") {
ChoiceRow(
options = listOf(
"账号同步" to (state.exploreInputMode == ExploreInputMode.ContentSource),
"视频链接" to (state.exploreInputMode == ExploreInputMode.VideoLink),
"上传视频" to (state.exploreInputMode == ExploreInputMode.UploadVideo),
"输入文字" to (state.exploreInputMode == ExploreInputMode.Text)
@@ -300,6 +301,7 @@ private fun ExploreTab(state: StoryForgeUiState, vm: StoryForgeViewModel, onPick
onSelect = { label ->
vm.setExploreInputMode(
when (label) {
"账号同步" -> ExploreInputMode.ContentSource
"视频链接" -> ExploreInputMode.VideoLink
"上传视频" -> ExploreInputMode.UploadVideo
else -> ExploreInputMode.Text
@@ -319,6 +321,93 @@ private fun ExploreTab(state: StoryForgeUiState, vm: StoryForgeViewModel, onPick
)
Spacer(modifier = Modifier.height(12.dp))
when (state.exploreInputMode) {
ExploreInputMode.ContentSource -> {
Text(
text = "适合导入抖音、B 站、小红书创作者账号主页。抖音 public 页面抓不到时,也可以把分享页链接和账号标识手工填进来。",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurface.copy(alpha = 0.7f)
)
Spacer(modifier = Modifier.height(10.dp))
ChoiceRow(
options = listOf(
"抖音" to (state.accountSyncPlatform == "抖音"),
"B站" to (state.accountSyncPlatform == "bilibili"),
"小红书" to (state.accountSyncPlatform == "小红书")
),
onSelect = { label ->
vm.updateAccountSyncPlatform(
when (label) {
"B站" -> "bilibili"
else -> label
}
)
}
)
Spacer(modifier = Modifier.height(10.dp))
OutlinedTextField(
value = state.accountSyncUrl,
onValueChange = vm::updateAccountSyncUrl,
modifier = Modifier.fillMaxWidth(),
label = { Text("账号主页或分享页链接") },
minLines = 2
)
Spacer(modifier = Modifier.height(10.dp))
OutlinedTextField(
value = state.accountSyncHandle,
onValueChange = vm::updateAccountSyncHandle,
modifier = Modifier.fillMaxWidth(),
label = { Text("账号标识(可选)") },
singleLine = true
)
Spacer(modifier = Modifier.height(10.dp))
OutlinedTextField(
value = state.accountSyncTitle,
onValueChange = vm::updateAccountSyncTitle,
modifier = Modifier.fillMaxWidth(),
label = { Text("任务标题(可选)") },
singleLine = true
)
Spacer(modifier = Modifier.height(10.dp))
OutlinedTextField(
value = state.accountSyncMaxItems,
onValueChange = vm::updateAccountSyncMaxItems,
modifier = Modifier.fillMaxWidth(),
label = { Text("最多拉取视频数1-20") },
singleLine = true
)
Spacer(modifier = Modifier.height(12.dp))
Row(
modifier = Modifier.fillMaxWidth(),
horizontalArrangement = Arrangement.spacedBy(12.dp)
) {
Row(
modifier = Modifier.weight(1f),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.SpaceBetween
) {
Text("跳过已存在视频", style = MaterialTheme.typography.bodySmall)
Switch(
checked = state.accountSyncSkipExisting,
onCheckedChange = vm::setAccountSyncSkipExisting
)
}
Row(
modifier = Modifier.weight(1f),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.SpaceBetween
) {
Text("自动触发分析", style = MaterialTheme.typography.bodySmall)
Switch(
checked = state.accountSyncAutoTriggerAnalysis,
onCheckedChange = vm::setAccountSyncAutoTriggerAnalysis
)
}
}
Spacer(modifier = Modifier.height(12.dp))
Button(onClick = vm::submitContentSourceSync, enabled = !state.busy) {
Text("同步账号内容")
}
}
ExploreInputMode.VideoLink -> {
OutlinedTextField(
value = state.videoUrl,

View File

@@ -26,6 +26,7 @@ enum class StoryForgeAuthMode {
}
enum class ExploreInputMode {
ContentSource,
VideoLink,
UploadVideo,
Text
@@ -72,6 +73,13 @@ data class StoryForgeUiState(
val createKnowledgeBaseName: String = "",
val createKnowledgeBaseDescription: String = "",
val exploreInputMode: ExploreInputMode = ExploreInputMode.VideoLink,
val accountSyncPlatform: String = "抖音",
val accountSyncHandle: String = "",
val accountSyncUrl: String = "",
val accountSyncTitle: String = "",
val accountSyncMaxItems: String = "5",
val accountSyncSkipExisting: Boolean = true,
val accountSyncAutoTriggerAnalysis: Boolean = true,
val videoUrl: String = "",
val videoTitle: String = "",
val textTitle: String = "",
@@ -155,6 +163,35 @@ class StoryForgeViewModel(application: Application) : AndroidViewModel(applicati
_state.value = _state.value.copy(videoUrl = value)
}
fun updateAccountSyncPlatform(value: String) {
_state.value = _state.value.copy(accountSyncPlatform = value)
}
fun updateAccountSyncHandle(value: String) {
_state.value = _state.value.copy(accountSyncHandle = value)
}
fun updateAccountSyncUrl(value: String) {
_state.value = _state.value.copy(accountSyncUrl = value)
}
fun updateAccountSyncTitle(value: String) {
_state.value = _state.value.copy(accountSyncTitle = value)
}
fun updateAccountSyncMaxItems(value: String) {
val digits = value.filter { it.isDigit() }
_state.value = _state.value.copy(accountSyncMaxItems = digits)
}
fun setAccountSyncSkipExisting(value: Boolean) {
_state.value = _state.value.copy(accountSyncSkipExisting = value)
}
fun setAccountSyncAutoTriggerAnalysis(value: Boolean) {
_state.value = _state.value.copy(accountSyncAutoTriggerAnalysis = value)
}
fun updateVideoTitle(value: String) {
_state.value = _state.value.copy(videoTitle = value)
}
@@ -463,6 +500,43 @@ class StoryForgeViewModel(application: Application) : AndroidViewModel(applicati
}
}
fun submitContentSourceSync() {
val current = state.value
if (current.accountSyncUrl.isBlank()) {
setError("请先输入账号主页链接")
return
}
val knowledgeBaseId = selectedKnowledgeBaseIdOrFallback()
if (knowledgeBaseId.isBlank()) {
setError("请先选择知识库")
return
}
val maxItems = current.accountSyncMaxItems.toIntOrNull()?.coerceIn(1, 20) ?: 5
runBusy(message = "正在创建账号同步任务...", task = {
repository.createContentSourceSyncJob(
platform = current.accountSyncPlatform.trim(),
handle = current.accountSyncHandle.trim(),
sourceUrl = current.accountSyncUrl.trim(),
title = current.accountSyncTitle.trim(),
knowledgeBaseId = knowledgeBaseId,
assistantId = current.selectedAssistantId,
analysisModelProfileId = preferredModelId(),
maxItems = maxItems,
skipExisting = current.accountSyncSkipExisting,
autoTriggerAnalysis = current.accountSyncAutoTriggerAnalysis
)
}) { job ->
appendTimeline("账号同步任务已创建: ${job.title}")
_state.value = state.value.copy(
accountSyncHandle = "",
accountSyncUrl = "",
accountSyncTitle = "",
accountSyncMaxItems = maxItems.toString()
)
afterJobCreated(job)
}
}
fun submitText() {
val current = state.value
if (current.textTitle.isBlank() || current.textContent.isBlank()) {

View File

@@ -1,5 +1,9 @@
FROM python:3.11-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

View File

@@ -48,6 +48,18 @@ class Database:
with self.session() as conn:
conn.execute(sql, params)
def table_exists(self, name: str) -> bool:
row = self.fetch_one(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?",
(name,),
)
return bool(row)
def column_exists(self, table: str, column: str) -> bool:
with self.session() as conn:
rows = conn.execute(f"PRAGMA table_info({table})").fetchall()
return any(row["name"] == column for row in rows)
def init_schema(self) -> None:
schema = """
CREATE TABLE IF NOT EXISTS accounts (
@@ -90,10 +102,10 @@ class Database:
CREATE TABLE IF NOT EXISTS knowledge_bases (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
project_id TEXT,
name TEXT NOT NULL,
description TEXT NOT NULL DEFAULT '',
fastgpt_dataset_id TEXT,
sync_status TEXT NOT NULL DEFAULT 'pending',
sync_status TEXT NOT NULL DEFAULT 'ready',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
FOREIGN KEY(user_id) REFERENCES accounts(id) ON DELETE CASCADE
@@ -108,7 +120,9 @@ class Database:
transcript_text TEXT NOT NULL DEFAULT '',
style_summary TEXT NOT NULL DEFAULT '',
combined_text TEXT NOT NULL DEFAULT '',
fastgpt_collection_id TEXT NOT NULL DEFAULT '',
analysis_json TEXT NOT NULL DEFAULT '{}',
storyboard_json TEXT NOT NULL DEFAULT '[]',
source_artifact_json TEXT NOT NULL DEFAULT '{}',
analysis_model_profile_id TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
@@ -118,11 +132,12 @@ class Database:
CREATE TABLE IF NOT EXISTS assistants (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
project_id TEXT,
name TEXT NOT NULL,
description TEXT NOT NULL DEFAULT '',
system_prompt TEXT NOT NULL DEFAULT '',
generation_goal TEXT NOT NULL DEFAULT '',
fastgpt_app_key TEXT NOT NULL DEFAULT '',
config_json TEXT NOT NULL DEFAULT '{}',
model_profile_id TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
@@ -140,19 +155,27 @@ class Database:
CREATE TABLE IF NOT EXISTS jobs (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
project_id TEXT,
parent_job_id TEXT,
assistant_id TEXT,
knowledge_base_id TEXT NOT NULL,
content_source_id TEXT,
source_type TEXT NOT NULL,
line_type TEXT NOT NULL DEFAULT 'analysis',
workflow_key TEXT NOT NULL DEFAULT '',
orchestrator TEXT NOT NULL DEFAULT 'n8n',
provider_name TEXT NOT NULL DEFAULT '',
provider_task_id TEXT NOT NULL DEFAULT '',
source_url TEXT,
title TEXT NOT NULL,
language TEXT NOT NULL DEFAULT 'auto',
status TEXT NOT NULL,
transcript_text TEXT NOT NULL DEFAULT '',
style_summary TEXT NOT NULL DEFAULT '',
fastgpt_collection_id TEXT NOT NULL DEFAULT '',
upload_status TEXT NOT NULL DEFAULT 'pending',
error TEXT NOT NULL DEFAULT '',
artifacts_json TEXT NOT NULL DEFAULT '{}',
result_json TEXT NOT NULL DEFAULT '{}',
analysis_model_profile_id TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
@@ -161,6 +184,42 @@ class Database:
FOREIGN KEY(knowledge_base_id) REFERENCES knowledge_bases(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS projects (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
name TEXT NOT NULL,
description TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
FOREIGN KEY(user_id) REFERENCES accounts(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS content_sources (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
project_id TEXT,
source_kind TEXT NOT NULL,
platform TEXT NOT NULL DEFAULT '',
handle TEXT NOT NULL DEFAULT '',
source_url TEXT NOT NULL DEFAULT '',
title TEXT NOT NULL DEFAULT '',
local_path TEXT NOT NULL DEFAULT '',
metadata_json TEXT NOT NULL DEFAULT '{}',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
FOREIGN KEY(user_id) REFERENCES accounts(id) ON DELETE CASCADE,
FOREIGN KEY(project_id) REFERENCES projects(id) ON DELETE SET NULL
);
CREATE TABLE IF NOT EXISTS job_events (
id TEXT PRIMARY KEY,
job_id TEXT NOT NULL,
event_type TEXT NOT NULL,
payload_json TEXT NOT NULL DEFAULT '{}',
created_at TEXT NOT NULL,
FOREIGN KEY(job_id) REFERENCES jobs(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS app_updates (
id INTEGER PRIMARY KEY AUTOINCREMENT,
platform TEXT NOT NULL,
@@ -179,3 +238,103 @@ class Database:
"""
with self.session() as conn:
conn.executescript(schema)
self.migrate_schema()
def migrate_schema(self) -> None:
table_columns: dict[str, dict[str, str]] = {
"knowledge_bases": {
"project_id": "TEXT",
},
"knowledge_documents": {
"analysis_json": "TEXT NOT NULL DEFAULT '{}'",
"storyboard_json": "TEXT NOT NULL DEFAULT '[]'",
"source_artifact_json": "TEXT NOT NULL DEFAULT '{}'",
},
"assistants": {
"project_id": "TEXT",
"config_json": "TEXT NOT NULL DEFAULT '{}'",
},
"jobs": {
"project_id": "TEXT",
"parent_job_id": "TEXT",
"content_source_id": "TEXT",
"line_type": "TEXT NOT NULL DEFAULT 'analysis'",
"workflow_key": "TEXT NOT NULL DEFAULT ''",
"orchestrator": "TEXT NOT NULL DEFAULT 'n8n'",
"provider_name": "TEXT NOT NULL DEFAULT ''",
"provider_task_id": "TEXT NOT NULL DEFAULT ''",
"result_json": "TEXT NOT NULL DEFAULT '{}'",
},
}
for table, columns in table_columns.items():
if not self.table_exists(table):
continue
for column, definition in columns.items():
if self.column_exists(table, column):
continue
self.execute(f"ALTER TABLE {table} ADD COLUMN {column} {definition}")
self.ensure_default_projects()
def ensure_default_projects(self) -> None:
if not self.table_exists("projects"):
return
accounts = self.fetch_all("SELECT id, username FROM accounts ORDER BY created_at ASC")
for account in accounts:
project = self.fetch_one(
"SELECT * FROM projects WHERE user_id = ? ORDER BY created_at ASC LIMIT 1",
(account["id"],),
)
if not project:
project_id = f"proj_{account['id']}"
now = utc_now()
self.execute(
"""
INSERT INTO projects (id, user_id, name, description, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?)
""",
(
project_id,
account["id"],
f"{account['username']} 默认项目",
"系统自动创建的默认项目",
now,
now,
),
)
project = self.fetch_one("SELECT * FROM projects WHERE id = ?", (project_id,))
if not project:
continue
if self.column_exists("knowledge_bases", "project_id"):
self.execute(
"""
UPDATE knowledge_bases
SET project_id = ?
WHERE user_id = ? AND (project_id IS NULL OR project_id = '')
""",
(project["id"], account["id"]),
)
if self.column_exists("assistants", "project_id"):
self.execute(
"""
UPDATE assistants
SET project_id = ?
WHERE user_id = ? AND (project_id IS NULL OR project_id = '')
""",
(project["id"], account["id"]),
)
if self.column_exists("jobs", "project_id"):
self.execute(
"""
UPDATE jobs
SET project_id = ?
WHERE user_id = ? AND (project_id IS NULL OR project_id = '')
""",
(project["id"], account["id"]),
)

File diff suppressed because it is too large Load Diff

View File

@@ -1,48 +0,0 @@
from __future__ import annotations
from typing import Any
import httpx
class FastGPTClient:
def __init__(self, *, base_url: str, dataset_api_key: str, timeout: float = 60.0) -> None:
self.base_url = base_url.rstrip("/")
self.dataset_api_key = dataset_api_key.strip()
self.timeout = timeout
@property
def enabled(self) -> bool:
return bool(self.base_url and self.dataset_api_key)
async def ensure_dataset(self, name: str, intro: str = "") -> dict[str, Any] | None:
if not self.enabled:
return None
payload = {"name": name, "intro": intro}
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.post(
f"{self.base_url}/api/core/dataset/create",
headers={"Authorization": f"Bearer {self.dataset_api_key}"},
json=payload,
)
response.raise_for_status()
return response.json().get("data") or response.json()
async def add_text_document(self, dataset_id: str, name: str, text: str) -> dict[str, Any] | None:
if not self.enabled or not dataset_id.strip():
return None
payload = {
"datasetId": dataset_id,
"type": "text",
"name": name,
"trainingType": "chunk",
"text": text,
}
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.post(
f"{self.base_url}/api/core/dataset/collection/create/text",
headers={"Authorization": f"Bearer {self.dataset_api_key}"},
json=payload,
)
response.raise_for_status()
return response.json().get("data") or response.json()

View File

@@ -0,0 +1,217 @@
from __future__ import annotations
import mimetypes
from pathlib import Path
from typing import Any
import httpx
def _join_url(base_url: str, path: str) -> str:
base = base_url.rstrip("/")
if path.startswith("http://") or path.startswith("https://"):
return path
return f"{base}/{path.lstrip('/')}"
def _unwrap_response(payload: Any) -> dict[str, Any]:
if not isinstance(payload, dict):
return {"value": payload}
if payload.get("success") is True and "data" in payload:
data = payload.get("data")
if isinstance(data, dict):
return data
return {"value": data}
return payload
class N8NClient:
def __init__(
self,
*,
base_url: str,
workflow_paths: dict[str, str],
shared_secret: str = "",
timeout: float = 60.0,
) -> None:
self.base_url = base_url.rstrip("/")
self.workflow_paths = workflow_paths
self.shared_secret = shared_secret.strip()
self.timeout = timeout
@property
def enabled(self) -> bool:
return bool(self.base_url)
async def trigger(self, workflow_key: str, payload: dict[str, Any]) -> dict[str, Any]:
workflow_path = self.workflow_paths.get(workflow_key, "").strip()
if not workflow_path:
raise ValueError(f"workflow path not configured for {workflow_key}")
try:
workflow_path = workflow_path.format(**payload)
except KeyError:
pass
headers: dict[str, str] = {}
if self.shared_secret:
headers["X-Orchestrator-Secret"] = self.shared_secret
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.post(
_join_url(self.base_url, workflow_path),
json=payload,
headers=headers,
)
response.raise_for_status()
if not response.content:
return {"triggered": True}
return _unwrap_response(response.json())
class CutVideoClient:
def __init__(
self,
*,
base_url: str,
api_key: str = "",
timeout: float = 120.0,
upload_timeout: float = 1800.0,
) -> None:
self.base_url = base_url.rstrip("/")
self.api_key = api_key.strip()
self.timeout = timeout
self.upload_timeout = upload_timeout
@property
def enabled(self) -> bool:
return bool(self.base_url)
def _headers(self) -> dict[str, str]:
headers: dict[str, str] = {}
if self.api_key:
headers["Authorization"] = f"Bearer {self.api_key}"
return headers
async def submit_job(self, payload: dict[str, Any]) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.post(
_join_url(self.base_url, "/api/jobs"),
json=payload,
headers=self._headers(),
)
response.raise_for_status()
return _unwrap_response(response.json())
async def upload_source_file(self, source_path: Path, *, folder_name: str = "") -> dict[str, Any]:
content_type = mimetypes.guess_type(source_path.name)[0] or "application/octet-stream"
headers = self._headers()
data = {"folder_name": folder_name} if folder_name else {}
async with httpx.AsyncClient(timeout=self.upload_timeout) as client:
with source_path.open("rb") as handle:
response = await client.post(
_join_url(self.base_url, "/api/uploads"),
data=data,
files={"files": (source_path.name, handle, content_type)},
headers=headers,
)
response.raise_for_status()
return _unwrap_response(response.json())
async def get_task(self, task_id: str) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.get(
_join_url(self.base_url, f"/api/tasks/{task_id}"),
headers=self._headers(),
)
response.raise_for_status()
return _unwrap_response(response.json())
async def get_run(self, run_id: str) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.get(
_join_url(self.base_url, f"/api/runs/{run_id}"),
headers=self._headers(),
)
response.raise_for_status()
return _unwrap_response(response.json())
class AsrHttpClient:
def __init__(
self,
*,
base_url: str,
transcribe_path: str = "/transcribe",
field_name: str = "wav",
timeout: float = 120.0,
) -> None:
self.base_url = base_url.rstrip("/")
self.transcribe_path = transcribe_path
self.field_name = field_name.strip() or "wav"
self.timeout = timeout
@property
def enabled(self) -> bool:
return bool(self.base_url)
async def transcribe_audio(self, audio_path: Path) -> dict[str, Any]:
content_type = mimetypes.guess_type(audio_path.name)[0] or "application/octet-stream"
async with httpx.AsyncClient(timeout=self.timeout) as client:
with audio_path.open("rb") as handle:
response = await client.post(
_join_url(self.base_url, self.transcribe_path),
files={self.field_name: (audio_path.name, handle, content_type)},
)
response.raise_for_status()
return _unwrap_response(response.json())
class HuobaoDramaClient:
def __init__(self, *, base_url: str, timeout: float = 180.0) -> None:
self.base_url = base_url.rstrip("/")
self.timeout = timeout
@property
def enabled(self) -> bool:
return bool(self.base_url)
async def create_drama(self, payload: dict[str, Any]) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.post(
_join_url(self.base_url, "/api/v1/dramas"),
json=payload,
)
response.raise_for_status()
return _unwrap_response(response.json())
async def generate_image(self, payload: dict[str, Any]) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.post(
_join_url(self.base_url, "/api/v1/images"),
json=payload,
)
response.raise_for_status()
return _unwrap_response(response.json())
async def get_image(self, image_id: str) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.get(
_join_url(self.base_url, f"/api/v1/images/{image_id}"),
)
response.raise_for_status()
return _unwrap_response(response.json())
async def generate_video(self, payload: dict[str, Any]) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.post(
_join_url(self.base_url, "/api/v1/videos"),
json=payload,
)
response.raise_for_status()
return _unwrap_response(response.json())
async def get_video(self, video_id: str) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.get(
_join_url(self.base_url, f"/api/v1/videos/{video_id}"),
)
response.raise_for_status()
return _unwrap_response(response.json())

File diff suppressed because it is too large Load Diff

View File

@@ -3,3 +3,4 @@ uvicorn[standard]==0.34.0
httpx==0.28.1
python-multipart==0.0.20
pydantic==2.11.1
yt-dlp

View File

@@ -0,0 +1,65 @@
#!/usr/bin/env bash
set -euo pipefail
LEGACY_CONTAINERS=(
storyforge-fastgpt-plugin
storyforge-sandbox
storyforge-pg
storyforge-minio
storyforge-redis
storyforge-mongo
)
LEGACY_NETWORK="storyforge-net"
COLLECTOR_HEALTH_URL="${COLLECTOR_HEALTH_URL:-http://127.0.0.1:8081/healthz}"
N8N_HEALTH_URL="${N8N_HEALTH_URL:-http://127.0.0.1:5670/healthz}"
APPLY="${APPLY:-0}"
log() {
printf '[cleanup] %s\n' "$*"
}
check_url() {
local url="$1"
curl -fsS "$url" >/dev/null
}
log "preflight: verifying StoryForge collector and n8n"
check_url "$COLLECTOR_HEALTH_URL"
check_url "$N8N_HEALTH_URL"
log "legacy containers:"
for container in "${LEGACY_CONTAINERS[@]}"; do
if docker ps -a --format '{{.Names}}' | grep -Fxq "$container"; then
status="$(docker inspect --format '{{.State.Status}}' "$container")"
printf ' - %s (%s)\n' "$container" "$status"
else
printf ' - %s (missing)\n' "$container"
fi
done
if [[ "$APPLY" != "1" ]]; then
log "dry run complete. Re-run with APPLY=1 to stop and remove legacy containers."
exit 0
fi
for container in "${LEGACY_CONTAINERS[@]}"; do
if docker ps -a --format '{{.Names}}' | grep -Fxq "$container"; then
log "removing $container"
docker rm -f "$container" >/dev/null
fi
done
if docker network inspect "$LEGACY_NETWORK" >/dev/null 2>&1; then
if [[ "$(docker network inspect "$LEGACY_NETWORK" --format '{{len .Containers}}')" == "0" ]]; then
log "removing empty network $LEGACY_NETWORK"
docker network rm "$LEGACY_NETWORK" >/dev/null
else
log "network $LEGACY_NETWORK still has attached containers; leaving it in place"
fi
fi
log "post-check: verifying StoryForge collector and n8n"
check_url "$COLLECTOR_HEALTH_URL"
check_url "$N8N_HEALTH_URL"
log "legacy FastGPT runtime cleanup completed"

View File

@@ -1,56 +1,30 @@
version: "3.9"
services:
mongo:
image: mongo:6
container_name: storyforge-mongo
restart: unless-stopped
ports:
- "27017:27017"
volumes:
- ./data/mongo:/data/db
vectorDB:
image: pgvector/pgvector:pg16
container_name: storyforge-pgvector
n8n:
image: ${N8N_IMAGE:-docker.n8n.io/n8nio/n8n:latest}
container_name: storyforge-n8n
restart: unless-stopped
environment:
POSTGRES_DB: ${POSTGRES_DB:-fastgpt}
POSTGRES_USER: ${POSTGRES_USER:-postgres}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres}
N8N_HOST: ${N8N_HOST:-0.0.0.0}
N8N_PORT: 5678
N8N_PROTOCOL: ${N8N_PROTOCOL:-http}
WEBHOOK_URL: ${WEBHOOK_URL:-http://127.0.0.1:5670/}
GENERIC_TIMEZONE: ${GENERIC_TIMEZONE:-Asia/Shanghai}
TZ: ${TZ:-Asia/Shanghai}
N8N_SECURE_COOKIE: ${N8N_SECURE_COOKIE:-false}
N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS: ${N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS:-false}
ports:
- "5432:5432"
- "5670:5678"
volumes:
- ./data/pg:/var/lib/postgresql/data
redis:
image: redis:7-alpine
container_name: storyforge-redis
restart: unless-stopped
ports:
- "6379:6379"
volumes:
- ./data/redis:/data
minio:
image: minio/minio:RELEASE.2025-02-07T23-21-09Z
container_name: storyforge-minio
restart: unless-stopped
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minioadmin}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-minioadmin}
ports:
- "9000:9000"
- "9001:9001"
volumes:
- ./data/minio:/data
- ./data/n8n:/home/node/.n8n
- ./n8n:/workspace/n8n:ro
collector:
build:
context: ./collector-service
container_name: storyforge-collector
restart: unless-stopped
depends_on:
- n8n
environment:
DATA_DIR: /data/collector
DATABASE_PATH: /data/collector/storyforge.db
@@ -58,40 +32,35 @@ services:
LOCAL_OPENAI_BASE_URL: ${LOCAL_OPENAI_BASE_URL:-http://host.docker.internal:8317/v1}
LOCAL_OPENAI_MODEL: ${LOCAL_OPENAI_MODEL:-GLM-5}
LOCAL_OPENAI_API_KEY: ${LOCAL_OPENAI_API_KEY:-}
FASTGPT_BASE_URL: ${FASTGPT_BASE_URL:-http://host.docker.internal:3000}
FASTGPT_DATASET_API_KEY: ${FASTGPT_DATASET_API_KEY:-}
N8N_BASE_URL: ${COLLECTOR_N8N_BASE_URL:-http://n8n:5678}
N8N_ANALYSIS_WEBHOOK_PATH: ${N8N_ANALYSIS_WEBHOOK_PATH:-/webhook/storyforge-analysis}
N8N_REAL_CUT_WEBHOOK_PATH: ${N8N_REAL_CUT_WEBHOOK_PATH:-/webhook/storyforge-real-cut}
N8N_AI_VIDEO_WEBHOOK_PATH: ${N8N_AI_VIDEO_WEBHOOK_PATH:-/webhook/storyforge-ai-video}
N8N_CONTENT_SOURCE_SYNC_WEBHOOK_PATH: ${N8N_CONTENT_SOURCE_SYNC_WEBHOOK_PATH:-/webhook/storyforge-content-source-sync}
ORCHESTRATOR_SHARED_SECRET: ${ORCHESTRATOR_SHARED_SECRET:-storyforge-local-secret}
CUTVIDEO_BASE_URL: ${CUTVIDEO_BASE_URL:-}
CUTVIDEO_API_KEY: ${CUTVIDEO_API_KEY:-}
CUTVIDEO_BASE_CONFIG: ${CUTVIDEO_BASE_CONFIG:-example.job.yaml}
CUTVIDEO_POLL_INTERVAL_SEC: ${CUTVIDEO_POLL_INTERVAL_SEC:-10}
CUTVIDEO_MAX_WAIT_SEC: ${CUTVIDEO_MAX_WAIT_SEC:-1800}
CUTVIDEO_UPLOAD_TIMEOUT_SEC: ${CUTVIDEO_UPLOAD_TIMEOUT_SEC:-1800}
HUOBAO_BASE_URL: ${HUOBAO_BASE_URL:-http://host.docker.internal:5678}
YTDLP_BIN: ${YTDLP_BIN:-yt-dlp}
FFMPEG_BIN: ${FFMPEG_BIN:-ffmpeg}
WHISPER_BIN: ${WHISPER_BIN:-}
WHISPER_MODEL: ${WHISPER_MODEL:-/data/collector/models/ggml-base.en.bin}
ASR_HTTP_BASE_URL: ${ASR_HTTP_BASE_URL:-}
ASR_HTTP_TRANSCRIBE_PATH: ${ASR_HTTP_TRANSCRIBE_PATH:-/transcribe}
ASR_HTTP_FIELD_NAME: ${ASR_HTTP_FIELD_NAME:-wav}
ASR_HTTP_TIMEOUT_SEC: ${ASR_HTTP_TIMEOUT_SEC:-120}
HUOBAO_POLL_INTERVAL_SEC: ${HUOBAO_POLL_INTERVAL_SEC:-10}
HUOBAO_MAX_WAIT_SEC: ${HUOBAO_MAX_WAIT_SEC:-900}
ports:
- "8081:8081"
volumes:
- ./data/collector:/data/collector
command: uvicorn app.main:app --host 0.0.0.0 --port 8081
fastgpt:
image: ghcr.io/labring/fastgpt:latest
container_name: storyforge-fastgpt
restart: unless-stopped
depends_on:
- mongo
- vectorDB
- redis
- minio
ports:
- "3000:3000"
sandbox:
image: ghcr.io/labring/fastgpt-sandbox:latest
container_name: storyforge-sandbox
restart: unless-stopped
fastgpt-plugin:
image: ghcr.io/labring/fastgpt-plugin:latest
container_name: storyforge-fastgpt-plugin
restart: unless-stopped
cli-proxy-api:
image: ${CLIPROXY_IMAGE:-storyforge/cli-proxy-api:patched}
container_name: storyforge-cliproxyapi

191
docs/AUDIT_2026-03-18.md Normal file
View File

@@ -0,0 +1,191 @@
# StoryForge 现状审计
日期2026-03-18
更新2026-03-20
## 结论
当前应以 `/Users/kris/code/StoryForge-gitea` 作为主工作区继续推进,而不是 `/Users/kris/code/Fastgpt`。后者更像一次不完整的导入快照,前者才是可持续开发的真实仓库。
## 现有功能归位
### 1. `collector-service` 之前承担的功能
- 账号注册、登录、审批
- 本地模型配置
- 知识库、智能体、任务管理
- 视频链接/上传视频/文本三类入口
- 下载器、ffmpeg、whisper.cpp 风格的本地处理调用
- Android OTA 查询/发布
### 2. FastGPT 实际承担的功能
- 仅承担“数据集/文档同步”的外部依赖角色
- 代码痕迹集中在:
- `collector-service/app/fastgpt.py`
- `docker-compose.yml`
- 若干 `fastgpt_*` 字段
结论FastGPT 并不是业务内核,适合迁移后整体删除。
### 3. n8n 适合接管的功能
- 任务触发
- 工作流分流
- 外部能力编排入口
- 任务执行顺序控制
不适合承载:
- 用户、项目、Agent、知识库、任务、历史记录的主数据
- 业务状态唯一真相源
结论:应采用“业务状态在 `collector-service`,流程编排在 `n8n`”的分层。
## 多用户与数据边界
当前已明确采用:
- `accounts`
- `projects`
- `knowledge_bases`
- `assistants`
- `content_sources`
- `jobs`
- `job_events`
推荐模型:`user + project`
理由:
- 只做 `user` 级隔离,会导致一个用户内部不同内容工作流难以再分边界
- `project` 可以自然承接“一个创作者方向 / 一个客户 / 一个账号矩阵 / 一个内容实验”
- `assistant``knowledge_base``job``content_source` 都能挂到 `project`,便于后续扩展协作空间
## 外部链路审计
### 1. 下载器
- 已存在,不需要重写
- 现阶段通过 `yt-dlp` 命令集成
- 账号级内容源同步同样复用 `yt-dlp --flat-playlist`,不额外维护抓取器
### 2. ASR
- 现有实现已部署,入口现已标准化为“两级优先级”:
- 优先调用 HTTP ASR 服务
- HTTP 不可用或返回空结果时,回退到 `whisper.cpp` 命令行
- 本次已按 `mac-whisper-service``/transcribe` 协议完成接入,并用任务 `job_e95f9b5579fd4c5aa40f04de611e9fd0` 验证 `artifacts.asr_backend=http`
- 进一步联调发现真实长视频转写耗时约 44 秒,因此 `collector``ASR_HTTP_TIMEOUT_SEC` 默认值已提升到 120 秒;本机 `mac-whisper-service` 运行时也需要把 `WHISPER_TIMEOUT_MS` 提升到 `120000`
- 修复后再次验证成功,任务 `job_bb405e2e878849e38c4bb31f7781e1e3` 已写入真实 HTTP ASR 文本并记录 `artifacts.asr_http_payload`
- `collector` 运行镜像已补上 `ffmpeg``yt-dlp`,避免容器内缺依赖导致音频抽取或下载失效
### 2.1 内容源账号同步
- 已新增 `content_source_sync_pipeline`
- 用户可通过 `POST /v2/pipelines/content-source-sync` 提交创作者账号 URL
- 后端会创建父任务,使用 `yt-dlp --flat-playlist` 抓取最近 N 条视频 URL再自动派生用户自己的 `video_link` 子分析任务
- `jobs.parent_job_id` 已加入数据模型,父子任务关系可持久化查询
- 已用 bilibili 账号 URL 联调验证:
- 父任务:`job_b02109cf9e8244fbb5b86f184a7c7574`
- 子任务:`job_7f169db61af441f8a7f186d03db2d91c``job_28c47774028441378a3974860c375ab7`
结论:账号级调度不再是空白能力,但目前只验证了 `bilibili` URL 形态,抖音 / 小红书仍需真链路核实。
### 3. Windows `cutvideo`
- 仓库:`/Users/kris/code/cutvideo`
- 具备清晰 API
- `POST /api/jobs`
- `POST /api/uploads`
- `GET /api/tasks/{task_id}`
- `GET /api/runs/{run_id}`
- 适合集成为“由 StoryForge 后端授权调用的局域网剪辑能力”
当前状态:
- StoryForge 已支持把 `upload_video` 或已完成的 `video_link` 源素材自动上传到 `cutvideo`
- `real-cut` 任务可直接传 `source_job_id`,由后端完成 staging 后再提交到剪辑服务
- Windows 机器已部署带 `POST /api/uploads``cutvideo` 版本,并完成局域网联调
### 3.1 `douyin` 工作台
- `collector-service` 已具备 `/v2/douyin/*` 工作台接口
- 已补充两类关键联调增强:
- 分享文案中的 URL 自动提取与归一化
- public 页面命中抖音反爬挑战时的显式诊断返回
- 真实 smoke 结果表明,纯 public 主页抓取会落到 `byted_acrawler` 挑战页,而不是正常 profile 数据页
- 同时,`manual_profile_payload + manual_work_payloads` 已验证可完成账号入库、分析报告生成、相似账号搜索和对标关系写入
- 现已新增浏览器辅助采集工具 `/Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture/capture_and_sync.mjs`
- 同目录现已新增本地控制台 `/Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture/control_panel.mjs`
- 该工具使用真实 Playwright Chromium 会话打开抖音页面,允许人工登录 / 过滑块后继续自动提取 `<script>` JSON、网络 JSON、视频详情页和创作者中心页数据
- 浏览器工具最终直接调用现有 `/v2/douyin/accounts/sync`,不新增第二套持久化模型
- 控制台模式已经支持“开始采集 -> 浏览器登录 -> 网页点继续 -> 自动同步”的点击式流程,并修复了 ready-file 提前点击的竞态
结论:`douyin` 方向不再是“接口存在但不可用”当前状态是“public 直抓受反爬限制,但人工采集兜底链已跑通”。
### 4. `huobao-drama`
- 旧改版位置:`/Users/kris/code/huobaoduanju/huobao-drama-master`
- 最新 upstream`/Users/kris/code/huobao-drama-upstream`
- 旧改版主要多了一套 `ad_workflow` 方向,和当前 StoryForge MVP 不完全对齐
- 最新版已具备:
- `POST /api/v1/dramas`
- `POST /api/v1/images`
- `GET /api/v1/images/{id}`
- `POST /api/v1/videos`
- `GET /api/v1/videos/{id}`
- `reference_mode=first_last`
本次真实联调里,旧改版为了兼容 `qnaigc` 需要补 4 个点:
- `pkg/image/openai_image_client.go`
- `application/services/image_generation_service.go`
- `pkg/video/openai_sora_client.go`
- `application/services/video_generation_service.go`
核对结果:
- 以上 4 个文件与本机 upstream 同名文件在补丁前没有明显结构分叉
- 当前差异基本就是 `qnaigc` 图片异步查询、Kling 视频 JSON 协议、结果 URL 解析、远程首尾帧 URL 保留这几处兼容逻辑
结论这批补丁是可移植补丁MVP 已在旧改版实例上验证通过;下一步应把同样补丁迁到最新版 `huobao-drama-upstream`,而不是继续在旧目录长期演进。
补充验证2026-03-20
- `/Users/kris/code/huobao-drama-upstream` 当前工作分支为 `codex/qnaigc-compat`
- 该分支已包含 qnaigc 图片异步查询、Kling 视频协议、结果 URL 解析、远程首尾帧保留等补丁
- 另外补了 `ResourceTransferService` 的 no-op MinIO 转存方法,当前 `go build ./...` 已可全量通过
- 使用复制自旧目录的 `config.yaml + drama_generator.db + data/storage` 在隔离目录启动了 upstream 实例,地址为 `http://127.0.0.1:5681`
- 上游实例健康检查通过,`POST /api/v1/dramas` 可正常创建剧本
- 新的图片和视频生成请求已能走到 provider 调用层,但当前复制出的 AI 配置凭证返回 `403 access denied for invalid user`
- 进一步在旧改版隔离实例 `http://127.0.0.1:5682` 上重放了 fresh 图片请求,返回同样的 `403 access denied for invalid user`
- 结论因此进一步收敛:当前 blocker 不是 upstream 回归,而是外部图片/视频凭证已失效
- 已在 `huobao-drama-upstream` 增加按服务类型的运行时覆盖能力,可用 `HUOBAO_TEXT_* / HUOBAO_IMAGE_* / HUOBAO_VIDEO_*` 环境变量接管数据库中的 AI 配置
- 已在 `huobao-drama-upstream` 固化 `scripts/run_storyforge_smoke.sh`,可自动复制旧库配置与数据、起隔离实例并校验 `/health`
结论更新:`huobao-drama-upstream` 的代码级兼容迁移已经完成,当前剩余 blocker 是外部图片/视频凭证失效,导致无法用“旧配置副本”继续 fresh 生成;但新的运行时 env 覆盖路径已经就位,后续补新 key 不需要再手改 SQLite。
## 当前已完成迁移面
- FastGPT 运行时依赖已从 `collector-service` 主代码中剥离
- 旧 FastGPT 运行残留容器 `storyforge-fastgpt-plugin / sandbox / pg / minio / redis / mongo` 已于 2026-03-20 实际下线并清理
- 数据库已支持 `project/content_source/job_events`
- `collector-service` 已增加:
- `n8n` 触发
- `cutvideo` 集成 client
- `huobao-drama` 集成 client
- 内部编排接口
- `docker-compose.yml` 已改为 `collector + n8n + cli-proxy-api`
- `n8n` 工作流导出文件已纳入仓库
- `collector-service` 的 live 运行态已回归到 `StoryForge-gitea` 自身源码构建,不再依赖 `/Users/kris/code/Fastgpt/collector-service/app` 的临时 bind mount
- `collector-service` 现已在 live `8081` 提供 `/v2/douyin/*` 接口,并保留原有 `real-cut / ai-video / content-source-sync` 路由
- Android Explore 页已补上“账号同步”入口,可直接创建内容源账号同步任务,并支持平台、主页链接、账号标识、最大抓取条数、跳过已存在、自动触发分析等参数
- Android 工作区缺失的 `com.aiglasses.app.data` 数据层已从同源代码补回,当前 `./gradlew :app:compileDebugKotlin``:app:assembleDebug` 均已通过,并产出 `app-debug.apk`
## 当前主要风险
1. 小红书账号级内容源还未做真实平台验证
2. `douyin` public 直抓仍受反爬限制,但现在已经有“真实浏览器 + 人工登录 + 自动提取 + 回写现有工作台”的可落地协作链
3. `huobao-drama-upstream` 已完成代码迁移并可编译,但 fresh smoke 受外部图片/视频凭证 `403 invalid user` 阻塞
4. Android 端目前已能完成 Debug APK 构建,但仍缺少真机安装和功能回归验证

View File

@@ -0,0 +1,97 @@
# StoryForge 分阶段实施计划
日期2026-03-18
## Phase 0: 审计与基线收拢
- 确认主工作区
- 识别 FastGPT 真实职责
- 识别多用户、多项目需要的主数据模型
- 对比 `huobao-drama` 旧改版与 upstream
- 审计 `cutvideo` 接口能力
状态:已完成
## Phase 1: 业务后端改造成主状态中心
- 引入 `projects`
- 引入 `content_sources`
- 引入 `job_events`
-`knowledge_bases / assistants / jobs` 全部 project 化
- 去掉 `collector-service` 中的 FastGPT 运行时逻辑
- 增加 `agents` 别名接口,统一 Agent 语义
状态:已完成首版
## Phase 2: n8n 接管流程编排
- 公共任务创建接口只负责建任务并触发工作流
- `n8n` 负责分发:
- `analysis_pipeline`
- `real_cut_pipeline`
- `ai_video_pipeline`
- 业务步骤落在 `collector-service` 内部接口,保证状态统一入库
状态:已完成首版
## Phase 3: 内容分析主线 MVP
- 支持文本
- 支持视频链接
- 支持上传视频
- 接下载器
- 接本地 ASR
- 接本地 LLM
- 产出:
- transcript
- style_summary
- analysis
- rewrite
- storyboards
状态:已完成首版
## Phase 4: 实拍自动剪辑主线 MVP
- 建立 `real_cut` 任务类型
- 通过 `n8n -> collector -> cutvideo` 调度 Windows 机器
- 记录 `task_id / run_id / 结果产物`
状态:已完成 API 级集成
待补:
- 用户上传素材到 Windows 侧的文件转运闭环
## Phase 5: AI 自动生成视频主线 MVP
- 建立 `ai_video` 任务类型
- 从分析结果或直接 brief 生成分镜
-`huobao-drama`
- 创建 drama
- 生成首帧
- 生成尾帧
- 基于首尾帧生成视频
- 结果回写任务
状态:已完成 API 级集成
## Phase 6: 删除 FastGPT 运行依赖
- 删除代码依赖
- 删除 compose 服务
- 删除环境变量
- 删除 README 说明
状态:已完成主仓库首版
## Phase 7: 联调与验证
- Python 语法检查
- Compose 配置检查
- `collector-service` 本地启动
- `n8n` workflow 导入
- Windows `cutvideo` 局域网调度
- `huobao-drama` 本机调用
状态:进行中

View File

@@ -0,0 +1,312 @@
# StoryForge 本地 / 局域网联调说明
日期2026-03-18
## 1. 准备 `.env`
复制:
```bash
cd /Users/kris/code/StoryForge-gitea
cp .env.example .env
```
至少确认这些变量:
- `N8N_BASE_URL=http://127.0.0.1:5670`,用于你在宿主机单独运行 `collector-service`
- `COLLECTOR_N8N_BASE_URL=http://n8n:5678`,用于 Docker 里的 `collector`
- `ORCHESTRATOR_SHARED_SECRET=storyforge-local-secret`
- `CUTVIDEO_BASE_URL=http://<windows-lan-ip>:7860`
- `CUTVIDEO_API_KEY=` 如果 Windows 服务启用了鉴权
- `HUOBAO_BASE_URL=http://127.0.0.1:5678`
- `WHISPER_BIN=` 指向你现有本地 ASR 可执行文件时填写
- `ASR_HTTP_BASE_URL=` 如果你已有常驻 ASR 服务,填写它的基地址
- `ASR_HTTP_TRANSCRIBE_PATH=/transcribe`
- `ASR_HTTP_FIELD_NAME=wav`
- `ASR_HTTP_TIMEOUT_SEC=120`
说明:
- 如果你单独重建 `collector`,要确保运行时仍带上 `CUTVIDEO_BASE_URL`,否则容器会退回空值
- `collector` 容器不要直接复用宿主机的 `N8N_BASE_URL=http://127.0.0.1:5670`,否则容器内会连回自己并导致 webhook 调度失败
- 当前已验证可用的 Windows `cutvideo` 地址是 `http://192.168.31.18:7860`
- 当前已验证可用的本机 HTTP ASR 入口是 `http://host.docker.internal:8088/transcribe`
- 如果你用的是本机 `mac-whisper-service`,建议同时以 `WHISPER_TIMEOUT_MS=120000` 启动,否则长视频会直接 504
## 2. 启动基础服务
```bash
cd /Users/kris/code/StoryForge-gitea
docker compose up -d --build
```
检查:
- `collector-service``http://127.0.0.1:8081/healthz`
- `n8n``http://127.0.0.1:5670`
- `cli-proxy-api``http://127.0.0.1:8317`
- 本机 `huobao-drama``http://127.0.0.1:5678/health`
## 3. 导入 n8n workflows
`n8n/workflows/` 导入:
- `storyforge-analysis.json`
- `storyforge-real-cut.json`
- `storyforge-ai-video.json`
- `storyforge-content-source-sync.json`
导入后:
- 检查每个 HTTP Request 节点的 `X-Orchestrator-Secret`
- 如果你改了 `.env` 的 secret这里必须同步
## 4. 登录与审批
默认超级管理员:
- 用户名:`kris`
- 密码:`Asd123456.`
新用户注册后,需要用超级管理员审批。
## 5. 内容分析链路验证
### 文本
调用 `POST /v2/explore/text`
预期:
- 任务创建成功
- `n8n` webhook 被触发
- 任务最终进入 `completed`
- 知识库文档里出现 transcript / style_summary / analysis / storyboards
已验证样例:
- `job_203bc8e9b20f4b1cbbc6cf7da79e46f4`
### 视频链接
调用 `POST /v2/explore/video-link`
前提:
- `yt-dlp` 可用
- `ffmpeg` 可用
- ASR 可调用
已验证样例:
- `job_bb405e2e878849e38c4bb31f7781e1e3` (`artifacts.asr_backend=http`)
### 上传视频
调用 `POST /v2/explore/upload-video`
预期与视频链接类似,但素材来源为本地上传
## 6. 内容源账号同步验证
调用 `POST /v2/pipelines/content-source-sync`
推荐最小请求体:
```json
{
"source_url": "https://space.bilibili.com/546195/video",
"platform": "bilibili",
"title": "Bilibili Creator Sync Smoke",
"max_items": 2,
"skip_existing": true,
"auto_trigger_analysis": true
}
```
预期:
- 创建一个 `content_source_sync` 父任务
- `n8n` 触发 `content_source_sync_pipeline`
- 父任务写回 `discovered_videos / child_job_ids / queued_job_ids`
- 子任务以 `parent_job_id` 挂到父任务下,并自动进入分析主线
已验证样例:
- 父任务:`job_b02109cf9e8244fbb5b86f184a7c7574`
- 子任务:`job_7f169db61af441f8a7f186d03db2d91c`
- 子任务:`job_28c47774028441378a3974860c375ab7`
## 6.1 `douyin` 账号工作台验证
基础接口:
- `POST /v2/douyin/accounts/sync`
- `POST /v2/douyin/accounts/{account_id}/analysis`
说明:
- `profile_url` 现在支持直接传分享文案,后端会自动提取里面的 URL
- 如果 public 页面命中抖音反爬挑战,接口会返回 `public_profile_anti_bot_challenge`
- 遇到挑战页时,继续可用的路径是 `manual_profile_payload``manual_work_payloads``manual_creator_pages`
已验证样例:
- public 页面 smoke返回 `public_profile_anti_bot_challenge`
- 手工导入账号:`dyacct_c2b62842b228406cb48f05fac16fdfdf`
- 手工账号分析报告:`dyreport_10d6b8d2d52a404192f54a3a05d44546`
- 相似账号搜索:`dysearch_c247b75db0df49429a1d127407fe4486`
- 对标关系:`dyrel_c8df266341e74237b99c880eb4b572d8`
浏览器辅助采集:
```bash
cd /Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture
npm install
npx playwright install chromium
npm run control-panel
```
浏览器打开:
```text
http://127.0.0.1:3618
```
控制台步骤:
1. 填写抖音主页链接和 StoryForge 账号
2. 如需查看采集结果,不用离开这个页面;下半部分 `Douyin Workbench` 会展示账号列表、Agent 结论、快照详情和对标结果
3. `作品工作台` 支持高分榜、最新榜和全部作品切换,并支持多种排序方式
4. 点击“自动分析高分作品”后,每条高分作品下会补齐商业判断、复刻建议、运营动作和风险提醒
2. 点击 `开始采集`
3. 在弹出的 Chromium 里登录或通过挑战页
4. 回到控制台点击 `已完成登录,继续采集`
5. 等待 `summary.json` 和可选的 `storyforge-sync-response.json`
命令行方式仍然保留:
```bash
cd /Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture
npm run capture -- \
--profile-url https://www.douyin.com/user/your_account \
--storyforge-username kris \
--storyforge-password 'Asd123456.'
```
说明:
- 脚本会打开真实 Chromium 会话,默认复用 `~/.storyforge/douyin-playwright` 登录态
- 如果出现扫码登录、滑块或挑战页,先在浏览器里人工完成,再回终端继续
- 脚本会保存 `profile-bundle.json``storyforge-sync-request.json` 和同步响应
- 当前已完成 headless 最小 smoke输出目录
- `/tmp/storyforge-douyin-capture-smoke/2026-03-20T06-49-37.705Z-storyforge_test_001`
- 当前已完成本地控制台 smoke输出目录
- `/Users/kris/code/StoryForge-gitea/output/playwright/douyin/control-panel/run-mmyzplxp-cw0o7q/2026-03-20T14-24-13.174Z-storyforge_test_001`
- `/Users/kris/code/StoryForge-gitea/output/playwright/douyin/control-panel/run-mmyzshsp-c6vdhi/2026-03-20T14-26-27.792Z-storyforge_test_001`
- 控制台模式已经修复“提前点击继续导致 ready 信号丢失”的竞态,早于等待点按钮也不会卡死
## 7. `cutvideo` 实拍剪辑链路验证
调用 `POST /v2/pipelines/real-cut`
当前 MVP 前提:
- 方式 A直接传 `input_dir`,它必须是 Windows `cutvideo` 机器可访问的目录
- 方式 B`source_job_id``collector-service` 会把 `upload_video` 或已完成的 `video_link` 源素材自动上传到 Windows `cutvideo`,再继续发起任务
- 如果走方式 B大文件上传超时由 `CUTVIDEO_UPLOAD_TIMEOUT_SEC` 控制
预期:
- 任务创建成功
- 如果用了 `source_job_id`,任务 `artifacts.cutvideo_upload` 会记录 Windows staging 结果
- `n8n` 调用 `collector-service` 内部 real-cut step
- 后端记录 `provider_task_id`
- 最终任务写回 `cutvideo_run`
已验证样例:
- `job_5ebd829c3f2144bca5c941183e75bdcd`
- `job_01a6f283cbda42e4ae692b268b811a50` (`source_job_id` 自动 staging本机 `cutvideo` 联调)
- Windows 返回 `task_id=8d8f4a0cd5d9`
- 运行目录 `20260318-093520-Windows cutvideo 联调样例`
## 8. `huobao-drama` AI 视频链路验证
调用 `POST /v2/pipelines/ai-video`
推荐方式:
- 先完成一个分析任务
- 再把该分析任务的 `source_job_id` 传给 AI 视频任务
预期:
- 创建 drama
- 每个分镜生成首帧、尾帧
- 每个分镜生成视频
- 最终 `job.result.rendered_scenes` 有完整结果
已验证样例:
- `job_01828c40377747cf914b51be360cc333`
- `provider_task_id=10`
- `video.task_id=qvideo-1380265978-1773799215825814468`
- 最终视频已回写到 `job.result.rendered_scenes[0].video.video_url`
补充说明2026-03-20
- `huobao-drama-upstream` 已在隔离目录用复制的旧配置和数据库起过实例,`/health` 正常
- fresh 图片/视频生成请求已能进入 provider 调用,但当前复制出的图片/视频凭证返回 `403 invalid user`
- 同样的 fresh 图片请求已在旧改版隔离实例 `http://127.0.0.1:5682` 上重放,结论一致,所以当前不是 upstream 回归问题
- `huobao-drama-upstream` 现在支持 `HUOBAO_TEXT_* / HUOBAO_IMAGE_* / HUOBAO_VIDEO_*` 运行时覆盖数据库里的 AI 配置
- `huobao-drama-upstream` 已新增 `/Users/kris/code/huobao-drama-upstream/scripts/run_storyforge_smoke.sh`,可自动复制旧目录配置和数据,在默认 `5681` 端口起隔离实例并校验 `/health`
- 如果你要重新验证 upstream fresh 生成,优先给 huobao 进程补这些环境变量,再复跑即可
推荐覆盖字段:
- `HUOBAO_TEXT_PROVIDER / BASE_URL / API_KEY / MODELS`
- `HUOBAO_IMAGE_PROVIDER / BASE_URL / API_KEY / MODELS`
- `HUOBAO_VIDEO_PROVIDER / BASE_URL / API_KEY / MODELS`
- 如需强制指定端点,还可补 `ENDPOINT / QUERY_ENDPOINT`
## 9. 当前已知卡点
- 抖音 public 页面直抓会命中反爬挑战;生产接入仍需要 cookie 或人工页面采集协助
- 小红书账号级内容源还未做真实平台验证
- `huobao-drama-upstream` 代码已迁移完成,但 fresh 生成仍受外部图片/视频凭证 `403 invalid user` 阻塞
## 10. 旧 FastGPT 残留清理
- 旧 FastGPT runtime 容器已在 2026-03-20 实际清理完成:
- `storyforge-fastgpt-plugin`
- `storyforge-sandbox`
- `storyforge-pg`
- `storyforge-minio`
- `storyforge-redis`
- `storyforge-mongo`
- 清理脚本已纳入仓库:
- `/Users/kris/code/StoryForge-gitea/deploy/cleanup_legacy_fastgpt_runtime.sh`
- 脚本会在清理前后校验:
- `http://127.0.0.1:8081/healthz`
- `http://127.0.0.1:5670/healthz`
## 11. Android 本地构建
如果你要在本机重新打 Android 包:
```bash
cd /Users/kris/code/StoryForge-gitea/android-app
./gradlew :app:assembleDebug
```
当前已验证结果:
- `:app:compileDebugKotlin` 通过
- `:app:assembleDebug` 通过
- APK 输出路径:
- `/Users/kris/code/StoryForge-gitea/android-app/app/build/outputs/apk/debug/app-debug.apk`
补充说明:
- 工作区根目录的 `.gitignore` 里保留了通用 `data/` 忽略规则,但已对 Android 源码目录 `android-app/app/src/main/java/com/aiglasses/app/data/` 做了白名单放行,避免误伤客户端代码

View File

@@ -0,0 +1,68 @@
# StoryForge MVP 状态
日期2026-03-18
更新2026-03-20
## 已跑通或已完成代码接通
- 多用户账号体系
- 审批机制
- `user -> project -> assistant / knowledge base / job / content source` 数据模型
- 文本 / 视频链接 / 上传视频 三类分析任务创建
- 内容源账号同步任务创建与子任务派发
- Android Explore 页已补上内容源账号同步入口
- Android `com.aiglasses.app.data` 数据层已补回,`compileDebugKotlin``assembleDebug` 已通过
- `n8n` 工作流导入、激活与触发接口
- 本地下载器调用
- 本地 `ffmpeg` / `whisper` 风格入口封装
- HTTP ASR 常驻服务入口绑定
- 本地大模型内容分析、二创文案、分镜生成
- Windows `cutvideo` API 调度与结果回写接口
- `upload_video -> source_job_id -> cutvideo` 自动 staging 闭环
- `collector` live 运行态已从临时源码挂载切回 `StoryForge-gitea` 正式镜像
- live `collector` 已挂出 `/v2/douyin/*` 能力并通过认证接口验证
- `douyin` 支持从分享文案中提取 `profile_url`,并在 public 页面命中抖音反爬挑战时返回明确诊断
- `douyin` 手工 payload 导入与账号分析链路已跑通
- `douyin` 浏览器辅助采集工具已接入,可用真实 Playwright Chromium 会话采集主页 / 视频页并直接调用现有 `/v2/douyin/accounts/sync`
- `douyin` 本地控制台已接入,可通过网页点击方式驱动浏览器辅助采集并查看最近运行结果
- 本机 `huobao-drama` API 调度、首尾帧生成、视频生成与结果回写接口
- FastGPT 运行时依赖删除
- 旧 FastGPT 运行残留容器已实际下线
## 已验证的真实任务
- 分析链路:`job_203bc8e9b20f4b1cbbc6cf7da79e46f4`
- HTTP ASR 分析链路:`job_e95f9b5579fd4c5aa40f04de611e9fd0`
- 账号级内容源同步链路:`job_b02109cf9e8244fbb5b86f184a7c7574`
- 账号级同步派生分析任务:`job_7f169db61af441f8a7f186d03db2d91c``job_28c47774028441378a3974860c375ab7`
- 长视频 HTTP ASR 超时修复后链路:`job_bb405e2e878849e38c4bb31f7781e1e3`
- 实拍剪辑链路:`job_5ebd829c3f2144bca5c941183e75bdcd`
- 实拍剪辑自动 staging 联调:`job_01a6f283cbda42e4ae692b268b811a50`
- AI 视频链路:`job_01828c40377747cf914b51be360cc333`
- Windows `cutvideo` 部署后联调:`job_5838515ed5c34679acd55a52cfcd424b`
- `douyin` 手工导入账号:`dyacct_c2b62842b228406cb48f05fac16fdfdf`
- `douyin` 账号分析报告:`dyreport_10d6b8d2d52a404192f54a3a05d44546`
- `douyin` 相似账号搜索:`dysearch_c247b75db0df49429a1d127407fe4486`
- `douyin` 对标关系:`dyrel_c8df266341e74237b99c880eb4b572d8`
- `huobao-upstream` 隔离 smoke 剧本:`drama_id=11` (`http://127.0.0.1:5681`)
- `huobao-upstream` 隔离 smoke 启动脚本:`/Users/kris/code/huobao-drama-upstream/scripts/run_storyforge_smoke.sh`
- Android Debug APK`/Users/kris/code/StoryForge-gitea/android-app/app/build/outputs/apk/debug/app-debug.apk`
- `douyin` 浏览器采集最小 smoke`/tmp/storyforge-douyin-capture-smoke/2026-03-20T06-49-37.705Z-storyforge_test_001`
- `douyin` 控制台 smoke`/Users/kris/code/StoryForge-gitea/output/playwright/douyin/control-panel/run-mmyzplxp-cw0o7q/2026-03-20T14-24-13.174Z-storyforge_test_001`
- `douyin` 控制台提前继续回归 smoke`/Users/kris/code/StoryForge-gitea/output/playwright/douyin/control-panel/run-mmyzshsp-c6vdhi/2026-03-20T14-26-27.792Z-storyforge_test_001`
## 尚未完全跑通
- 小红书账号级内容源还未做真实平台验证
- `douyin` public 主页直抓会命中 `public_profile_anti_bot_challenge`;当前已验证手工 payload 导入、分析、相似账号搜索和对标关系可作为可用兜底路径
- `douyin` 浏览器辅助采集已经能真实输出 `profile-bundle.json / storyforge-sync-request.json`,但要拿到有效主页数据仍需要用户在浏览器里完成登录或挑战校验
- `douyin` 控制台点击流已可用,但它仍然依赖本机可打开 Chromium 的环境,不适合放进纯 Docker 容器内部跑 GUI
- `huobao-upstream` 已能全量编译;并且旧改版隔离实例也已重放确认,当前 fresh 生成被外部图片/视频凭证统一返回 `403 invalid user`
- `huobao-upstream` 已新增 `HUOBAO_TEXT_* / HUOBAO_IMAGE_* / HUOBAO_VIDEO_*` 运行时覆盖能力,后续补新 key 可直接接管数据库配置
- Android Debug 包已可本地构建,但尚未完成真机安装验证
## 下一步优先级
1. 更新 `huobao` 可用图片/视频凭证后,用新的 env 覆盖能力对 upstream 版补一轮完整 `drama -> images -> video` fresh smoke
2. 补抖音真实账号的 cookie / 手工页面采集联调,以及小红书账号级验证
3.`collector` live 切换结果和部署回滚说明继续固化到仓库

30
n8n/README.md Normal file
View File

@@ -0,0 +1,30 @@
# n8n Workflows
本目录保存 StoryForge 的工作流导出文件,避免流程只存在于 n8n UI。
## 工作流
- `workflows/storyforge-analysis.json`:内容分析主线
- `workflows/storyforge-real-cut.json`Windows `cutvideo` 调度主线
- `workflows/storyforge-ai-video.json``huobao-drama` AI 生成视频主线
- `workflows/storyforge-content-source-sync.json`:内容源账号同步与批量分析派发主线
## 约定
- 工作流内部默认通过 `http://collector:8081` 调用 `collector-service`
- 内部调用头部使用 `X-Orchestrator-Secret: storyforge-local-secret`
- 如果你修改了 `.env` 里的 `ORCHESTRATOR_SHARED_SECRET`,导入工作流后需要同步更新对应 HTTP Request 节点
## 导入
1. 先执行 `docker compose up -d n8n collector`
2. 打开 `http://127.0.0.1:5670`
3. 从 UI 导入本目录下的 4 个 JSON
4. 激活工作流
## Webhook 路径
- `/webhook/storyforge-analysis`
- `/webhook/storyforge-real-cut`
- `/webhook/storyforge-ai-video`
- `/webhook/storyforge-content-source-sync`

View File

@@ -0,0 +1,70 @@
{
"name": "StoryForge AI Video Pipeline",
"nodes": [
{
"parameters": {
"httpMethod": "POST",
"path": "storyforge-ai-video",
"responseMode": "onReceived",
"options": {}
},
"id": "aivideo-webhook",
"name": "AI Video Webhook",
"type": "n8n-nodes-base.webhook",
"typeVersion": 2,
"position": [
220,
300
],
"webhookId": "storyforge-ai-video"
},
{
"parameters": {
"method": "POST",
"url": "={{'http://collector:8081/internal/jobs/steps/ai-video/render?job_id=' + ($json.body.job_id || $json.body.jobId)}}",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "X-Orchestrator-Secret",
"value": "storyforge-local-secret"
}
]
},
"options": {
"timeout": 3600000
}
},
"id": "aivideo-runner",
"name": "Run AI Video Step",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
520,
300
]
}
],
"connections": {
"AI Video Webhook": {
"main": [
[
{
"node": "Run AI Video Step",
"type": "main",
"index": 0
}
]
]
},
"Run AI Video Step": {
"main": [
[]
]
}
},
"active": false,
"settings": {},
"pinData": {},
"versionId": "storyforge-ai-video-v1"
}

View File

@@ -0,0 +1,70 @@
{
"name": "StoryForge Analysis Pipeline",
"nodes": [
{
"parameters": {
"httpMethod": "POST",
"path": "storyforge-analysis",
"responseMode": "onReceived",
"options": {}
},
"id": "analysis-webhook",
"name": "Analysis Webhook",
"type": "n8n-nodes-base.webhook",
"typeVersion": 2,
"position": [
220,
300
],
"webhookId": "storyforge-analysis"
},
{
"parameters": {
"method": "POST",
"url": "={{'http://collector:8081/internal/jobs/steps/analyze?job_id=' + ($json.body.job_id || $json.body.jobId)}}",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "X-Orchestrator-Secret",
"value": "storyforge-local-secret"
}
]
},
"options": {
"timeout": 600000
}
},
"id": "analysis-runner",
"name": "Run Analysis Step",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
520,
300
]
}
],
"connections": {
"Analysis Webhook": {
"main": [
[
{
"node": "Run Analysis Step",
"type": "main",
"index": 0
}
]
]
},
"Run Analysis Step": {
"main": [
[]
]
}
},
"active": false,
"settings": {},
"pinData": {},
"versionId": "storyforge-analysis-v1"
}

View File

@@ -0,0 +1,70 @@
{
"name": "StoryForge Content Source Sync",
"nodes": [
{
"parameters": {
"httpMethod": "POST",
"path": "storyforge-content-source-sync",
"responseMode": "onReceived",
"options": {}
},
"id": "content-source-sync-webhook",
"name": "Content Source Sync Webhook",
"type": "n8n-nodes-base.webhook",
"typeVersion": 2,
"position": [
220,
300
],
"webhookId": "storyforge-content-source-sync"
},
{
"parameters": {
"method": "POST",
"url": "={{'http://collector:8081/internal/jobs/steps/content-source-sync?job_id=' + ($json.body.job_id || $json.body.jobId)}}",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "X-Orchestrator-Secret",
"value": "storyforge-local-secret"
}
]
},
"options": {
"timeout": 600000
}
},
"id": "content-source-sync-runner",
"name": "Run Content Source Sync Step",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
540,
300
]
}
],
"connections": {
"Content Source Sync Webhook": {
"main": [
[
{
"node": "Run Content Source Sync Step",
"type": "main",
"index": 0
}
]
]
},
"Run Content Source Sync Step": {
"main": [
[]
]
}
},
"active": false,
"settings": {},
"pinData": {},
"versionId": "storyforge-content-source-sync-v1"
}

View File

@@ -0,0 +1,70 @@
{
"name": "StoryForge Real Cut Pipeline",
"nodes": [
{
"parameters": {
"httpMethod": "POST",
"path": "storyforge-real-cut",
"responseMode": "onReceived",
"options": {}
},
"id": "realcut-webhook",
"name": "Real Cut Webhook",
"type": "n8n-nodes-base.webhook",
"typeVersion": 2,
"position": [
220,
300
],
"webhookId": "storyforge-real-cut"
},
{
"parameters": {
"method": "POST",
"url": "={{'http://collector:8081/internal/jobs/steps/real-cut/run?job_id=' + ($json.body.job_id || $json.body.jobId)}}",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "X-Orchestrator-Secret",
"value": "storyforge-local-secret"
}
]
},
"options": {
"timeout": 3600000
}
},
"id": "realcut-runner",
"name": "Run Real Cut Step",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
520,
300
]
}
],
"connections": {
"Real Cut Webhook": {
"main": [
[
{
"node": "Run Real Cut Step",
"type": "main",
"index": 0
}
]
]
},
"Run Real Cut Step": {
"main": [
[]
]
}
},
"active": false,
"settings": {},
"pinData": {},
"versionId": "storyforge-real-cut-v1"
}

View File

@@ -1,7 +1,6 @@
#!/bin/sh
set -eu
cd /Users/kris/code/StoryForge/collector-service
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --host 0.0.0.0 --port 8081 --reload
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
"$ROOT/scripts/start_business.sh"

39
scripts/cleanup_debug_ui.sh Executable file
View File

@@ -0,0 +1,39 @@
#!/bin/sh
set -eu
PORT="${1:-3618}"
SCRIPT_PATH="/Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture/control_panel.mjs"
lsof -tiTCP:"$PORT" -sTCP:LISTEN | xargs -r kill || true
osascript -e 'tell application "Terminal"' \
-e 'if it is running then' \
-e 'repeat with w in windows' \
-e 'set shouldClose to false' \
-e 'repeat with t in tabs of w' \
-e 'try' \
-e 'set tabText to contents of t' \
-e 'if tabText contains "'"$SCRIPT_PATH"'" then set shouldClose to true' \
-e 'end try' \
-e 'end repeat' \
-e 'if shouldClose then close w saving no' \
-e 'end repeat' \
-e 'end if' \
-e 'end tell' >/dev/null 2>&1 || true
for app in "Google Chrome" "Brave Browser" "Arc" "Safari"; do
osascript -e 'try' \
-e 'tell application "'"$app"'"' \
-e 'repeat with w in windows' \
-e 'repeat with i from (count of tabs of w) to 1 by -1' \
-e 'try' \
-e 'set tabUrl to URL of tab i of w' \
-e 'if tabUrl contains "127.0.0.1:'"$PORT"'" then close tab i of w' \
-e 'end try' \
-e 'end repeat' \
-e 'end repeat' \
-e 'end tell' \
-e 'end try' >/dev/null 2>&1 || true
done
echo "debug ui cleaned: port $PORT"

View File

@@ -0,0 +1,75 @@
# Douyin Browser Capture
This tool drives a real Playwright Chromium session, lets a human log into Douyin, captures the loaded profile and work pages, and can sync the captured bundle into StoryForge's existing `/v2/douyin/accounts/sync` endpoint.
## Install
```bash
cd /Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture
npm install
npx playwright install chromium
```
## Run
```bash
cd /Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture
npm run capture -- \
--profile-url https://www.douyin.com/user/your_account \
--storyforge-username kris \
--storyforge-password 'Asd123456.'
```
The browser uses a persistent state directory under `~/.storyforge/douyin-playwright`, so Douyin login can survive between runs.
## Control Panel
If you do not want to remember CLI arguments, start the local control panel:
```bash
cd /Users/kris/code/StoryForge-gitea/scripts/douyin-browser-capture
npm run control-panel
```
Then open [http://127.0.0.1:3618](http://127.0.0.1:3618) and use this flow:
1. Fill in the Douyin profile URL and StoryForge credentials.
2. Click `开始采集`.
3. A real Chromium window opens. Log into Douyin and solve any captcha there.
4. Return to the control panel and click `已完成登录,继续采集`.
5. Wait for `summary.json` and the optional StoryForge sync result.
The control panel stores each run under:
`/Users/kris/code/StoryForge-gitea/output/playwright/douyin/control-panel`
## What it captures
- current profile page JSON blobs extracted from `<script>` tags
- selected window globals such as `__INITIAL_STATE__`
- relevant JSON network responses
- creator-center pages using the same logged-in browser context
- a limited number of video detail pages linked from the profile
## Output
Default output directory:
`/Users/kris/code/StoryForge-gitea/output/playwright/douyin`
Each run writes:
- `profile-bundle.json`
- `creator-*.json`
- `video-*.json`
- `storyforge-sync-request.json`
- `storyforge-sync-response.json` when sync is enabled
- `summary.json`
## Notes
- This is designed as a browser-assisted capture flow, not a fully headless anti-bot bypass.
- If Douyin shows a slider or challenge page, solve it manually in the opened browser window and then continue.
- Use `--no-sync` if you only want to save a local bundle for inspection.
- Use `--ready-file /tmp/storyforge-ready.signal` if you want another process or webpage to decide when capture continues.
- Creator-center pages belong to the currently logged-in Douyin account. StoryForge now treats them as supplemental evidence by default and will not let them overwrite the target profile unless you explicitly pass `--allow-creator-center-fallback`.

View File

@@ -0,0 +1,855 @@
#!/usr/bin/env node
import fs from "node:fs/promises";
import { execFileSync } from "node:child_process";
import os from "node:os";
import path from "node:path";
import process from "node:process";
import readline from "node:readline/promises";
import { stdin as input, stdout as output } from "node:process";
import { chromium } from "playwright";
const DEFAULT_CREATOR_CENTER_URLS = [
"https://creator.douyin.com/creator-micro/home",
"https://creator.douyin.com/creator-micro/data",
"https://creator.douyin.com/creator-micro/content/manage"
];
const DEFAULT_OUTPUT_DIR = "/Users/kris/code/StoryForge-gitea/output/playwright/douyin";
const DEFAULT_STATE_DIR = path.join(os.homedir(), ".storyforge", "douyin-playwright");
const DEFAULT_BACKEND_URL = "http://127.0.0.1:8081";
const JSON_CAPTURE_LIMIT = 1_500_000;
const SCRIPT_SCAN_LIMIT = 2_000_000;
const WAIT_AFTER_NAV_MS = 4_000;
const RESPONSE_READ_TIMEOUT_MS = 2_000;
const PYTHON_HTTP_BRIDGE = `
import json
import sys
import urllib.error
import urllib.request
url, method, headers_json, body_mode, body_value = sys.argv[1:6]
headers = json.loads(headers_json)
body = None
if body_mode == "text":
body = body_value.encode("utf-8")
elif body_mode == "path":
with open(body_value, "rb") as handle:
body = handle.read()
request = urllib.request.Request(url, data=body, headers=headers, method=method)
try:
with urllib.request.urlopen(request, timeout=120) as response:
raw = response.read().decode("utf-8", "replace")
try:
payload = json.loads(raw) if raw else None
except Exception:
payload = {"raw": raw}
print(json.dumps({"status": response.status, "data": payload}, ensure_ascii=False))
except urllib.error.HTTPError as error:
raw = error.read().decode("utf-8", "replace")
try:
payload = json.loads(raw) if raw else None
except Exception:
payload = {"raw": raw}
print(json.dumps({"status": error.code, "data": payload}, ensure_ascii=False))
except Exception as error:
print(json.dumps({"status": 599, "data": {"raw": str(error)}}, ensure_ascii=False))
`;
function printHelp() {
console.log(`StoryForge Douyin Browser Capture
Usage:
node capture_and_sync.mjs --profile-url <douyin-profile-url> [options]
Core options:
--profile-url <url> Douyin profile URL to capture
--backend-url <url> StoryForge collector base URL (default: ${DEFAULT_BACKEND_URL})
--output-dir <dir> Capture output directory (default: ${DEFAULT_OUTPUT_DIR})
--state-dir <dir> Persistent browser state dir (default: ${DEFAULT_STATE_DIR})
--max-videos <n> Max video detail pages to capture (default: 4)
--scroll-count <n> Scroll times on profile page (default: 5)
--wait-ms <n> Wait after each navigation in ms (default: ${WAIT_AFTER_NAV_MS})
--ready-file <path> Wait for this file to appear instead of terminal prompt
StoryForge auth:
--storyforge-token <token> Existing StoryForge bearer token
--storyforge-username <name> Login username for StoryForge
--storyforge-password <pass> Login password for StoryForge
Mode flags:
--headless Run browser headless
--skip-login-prompt Do not pause for manual login / captcha completion
--no-sync Capture only, do not import into StoryForge
--no-creator-center Skip creator-center page capture
--allow-creator-center-fallback
Allow creator-center identity to replace a missing public profile
--note <text> Discovery note saved into StoryForge
Examples:
npm run capture -- \\
--profile-url https://www.douyin.com/user/your_account \\
--storyforge-username kris --storyforge-password 'Asd123456.'
npm run capture -- \\
--profile-url https://www.douyin.com/user/your_account \\
--storyforge-token <token> --headless --skip-login-prompt --no-creator-center
`);
}
function parseArgs(argv) {
const options = {
backendUrl: DEFAULT_BACKEND_URL,
outputDir: DEFAULT_OUTPUT_DIR,
stateDir: DEFAULT_STATE_DIR,
maxVideos: 4,
scrollCount: 5,
waitMs: WAIT_AFTER_NAV_MS,
headless: false,
manualPrompt: true,
syncEnabled: true,
creatorCenterEnabled: true,
allowCreatorCenterFallback: false,
creatorCenterUrls: [...DEFAULT_CREATOR_CENTER_URLS],
note: "",
profileUrl: "",
readyFile: "",
storyforgeToken: "",
storyforgeUsername: "",
storyforgePassword: ""
};
const requireValue = (index, flag) => {
const value = argv[index + 1];
if (!value || value.startsWith("--")) {
throw new Error(`Missing value for ${flag}`);
}
return value;
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
switch (arg) {
case "--help":
case "-h":
options.help = true;
break;
case "--profile-url":
options.profileUrl = requireValue(index, arg);
index += 1;
break;
case "--backend-url":
options.backendUrl = requireValue(index, arg);
index += 1;
break;
case "--output-dir":
options.outputDir = requireValue(index, arg);
index += 1;
break;
case "--state-dir":
options.stateDir = requireValue(index, arg);
index += 1;
break;
case "--max-videos":
options.maxVideos = Number.parseInt(requireValue(index, arg), 10);
index += 1;
break;
case "--scroll-count":
options.scrollCount = Number.parseInt(requireValue(index, arg), 10);
index += 1;
break;
case "--wait-ms":
options.waitMs = Number.parseInt(requireValue(index, arg), 10);
index += 1;
break;
case "--ready-file":
options.readyFile = requireValue(index, arg);
index += 1;
break;
case "--storyforge-token":
options.storyforgeToken = requireValue(index, arg);
index += 1;
break;
case "--storyforge-username":
options.storyforgeUsername = requireValue(index, arg);
index += 1;
break;
case "--storyforge-password":
options.storyforgePassword = requireValue(index, arg);
index += 1;
break;
case "--note":
options.note = requireValue(index, arg);
index += 1;
break;
case "--headless":
options.headless = true;
break;
case "--skip-login-prompt":
options.manualPrompt = false;
break;
case "--no-sync":
options.syncEnabled = false;
break;
case "--no-creator-center":
options.creatorCenterEnabled = false;
break;
case "--allow-creator-center-fallback":
options.allowCreatorCenterFallback = true;
break;
default:
throw new Error(`Unknown argument: ${arg}`);
}
}
return options;
}
function sanitizeName(value) {
return String(value || "capture")
.replace(/[^a-zA-Z0-9._-]+/g, "-")
.replace(/-+/g, "-")
.replace(/^-|-$/g, "")
.slice(0, 80) || "capture";
}
async function ensureDir(dir) {
await fs.mkdir(dir, { recursive: true });
}
function nowStamp() {
return new Date().toISOString().replace(/[:]/g, "-");
}
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function navigateAndSettle(page, url, waitMs) {
await page.goto(url, { waitUntil: "commit", timeout: 30_000 }).catch(() => null);
await page.waitForLoadState("domcontentloaded", { timeout: 15_000 }).catch(() => {});
await sleep(waitMs);
}
async function maybePrompt(message, enabled, readyFile = "") {
if (!enabled) {
return;
}
if (readyFile) {
console.error(`${message}\nWaiting for ready file: ${readyFile}`);
await waitForReadyFile(readyFile);
return;
}
const rl = readline.createInterface({ input, output });
try {
await rl.question(`${message}\nPress Enter to continue... `);
} finally {
rl.close();
}
}
async function waitForReadyFile(filePath) {
await ensureDir(path.dirname(filePath));
while (true) {
try {
await fs.access(filePath);
return;
} catch {
await sleep(600);
}
}
}
function uniqueStrings(values) {
const seen = new Set();
const output = [];
for (const value of values) {
const item = String(value || "").trim();
if (!item || seen.has(item)) {
continue;
}
seen.add(item);
output.push(item);
}
return output;
}
function looksLikeRelevantJsonUrl(url) {
const lower = url.toLowerCase();
return (
lower.includes("douyin.com/aweme") ||
lower.includes("douyin.com/web/api") ||
lower.includes("douyin.com/creator") ||
lower.includes("douyin.com/user") ||
lower.includes("creator.douyin.com") ||
lower.includes("iesdouyin.com")
);
}
function findJsonEnd(text, start) {
const opening = text[start];
const closing = opening === "{" ? "}" : "]";
let depth = 0;
let inString = false;
let escaped = false;
for (let index = start; index < text.length; index += 1) {
const char = text[index];
if (inString) {
if (escaped) {
escaped = false;
} else if (char === "\\") {
escaped = true;
} else if (char === "\"") {
inString = false;
}
continue;
}
if (char === "\"") {
inString = true;
continue;
}
if (char === opening) {
depth += 1;
continue;
}
if (char === closing) {
depth -= 1;
if (depth === 0) {
return index + 1;
}
}
}
return -1;
}
async function createResponseCapture(page) {
const records = [];
const seen = new Set();
const pending = [];
const listener = (response) => {
const promise = (async () => {
try {
const url = response.url();
const headers = response.headers();
const contentType = String(headers["content-type"] || "").toLowerCase();
if (!contentType.includes("json") && !looksLikeRelevantJsonUrl(url)) {
return;
}
const key = `${response.request().method()} ${url}`;
if (seen.has(key)) {
return;
}
const text = await Promise.race([
response.text(),
sleep(RESPONSE_READ_TIMEOUT_MS).then(() => {
throw new Error("response read timeout");
})
]);
if (!text || text.length > JSON_CAPTURE_LIMIT) {
return;
}
let payload = null;
try {
payload = JSON.parse(text);
} catch {
return;
}
seen.add(key);
records.push({
url,
method: response.request().method(),
status: response.status(),
payload
});
} catch {
// Ignore network capture failures; page-level capture is still useful.
}
})();
pending.push(promise);
};
page.on("response", listener);
return {
records,
async stop() {
page.off("response", listener);
await Promise.race([
Promise.allSettled(pending),
sleep(RESPONSE_READ_TIMEOUT_MS + 500)
]);
return records;
}
};
}
function extractJsonObjectsFromText(text) {
const candidates = [text];
const seen = new Set();
const results = [];
for (const candidate of candidates) {
const snippet = String(candidate || "").slice(0, SCRIPT_SCAN_LIMIT);
for (let index = 0; index < snippet.length; index += 1) {
const char = snippet[index];
if (char !== "{" && char !== "[") {
continue;
}
const end = findJsonEnd(snippet, index);
if (end <= index) {
continue;
}
try {
const parsed = JSON.parse(snippet.slice(index, end));
const marker = JSON.stringify(parsed);
if (seen.has(marker)) {
continue;
}
seen.add(marker);
results.push(parsed);
if (results.length >= 50) {
return results;
}
} catch {
// Keep scanning.
}
}
}
return results;
}
function extractScriptPayloads(html) {
const results = [];
const seen = new Set();
const regex = /<script([^>]*)>([\s\S]*?)<\/script>/gi;
let match = null;
while ((match = regex.exec(html)) !== null) {
const attrs = match[1] || "";
const content = match[2] || "";
const idMatch = attrs.match(/id=["']([^"']+)["']/i);
const scriptId = idMatch ? idMatch[1] : "";
for (const payload of extractJsonObjectsFromText(content.trim())) {
const marker = JSON.stringify(payload);
if (seen.has(marker)) {
continue;
}
seen.add(marker);
results.push({ script_id: scriptId, payload });
}
}
return results;
}
async function collectWindowGlobals(page) {
return page.evaluate(() => {
const globalNames = [
"__INITIAL_STATE__",
"__NEXT_DATA__",
"__ROUTER_DATA__",
"SIGI_STATE",
"__APOLLO_STATE__"
];
const result = {};
for (const name of globalNames) {
const value = globalThis[name];
if (value === undefined) {
continue;
}
try {
result[name] = JSON.parse(JSON.stringify(value));
} catch {
// Skip non-serializable globals.
}
}
return result;
});
}
async function collectVideoLinks(page) {
const hrefs = await page.evaluate(() => {
return Array.from(document.querySelectorAll("a[href]"))
.map((node) => node.getAttribute("href") || "")
.filter(Boolean);
});
return uniqueStrings(
hrefs
.map((href) => {
if (href.startsWith("//")) {
return `https:${href}`;
}
if (href.startsWith("/")) {
return `https://www.douyin.com${href}`;
}
return href;
})
.filter((href) => href.includes("/video/"))
);
}
async function clickFirstVisible(page, selectors) {
for (const selector of selectors) {
const locator = page.locator(selector).first();
try {
if (await locator.isVisible({ timeout: 1000 })) {
await locator.click({ timeout: 1000 });
return true;
}
} catch {
// Try next selector.
}
}
return false;
}
function escapeRegExp(value) {
return String(value || "").replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
function decodeEscapedUrl(value) {
return String(value || "")
.replace(/\\u002F/g, "/")
.replace(/\\\//g, "/")
.replace(/&amp;/g, "&");
}
async function resolveCreatorPrefetchUrl(page) {
const current = new URL(page.url());
const html = await page.content();
const escapedPath = escapeRegExp(current.pathname);
const mapped = html.match(
new RegExp(`"${escapedPath}"\\s*:\\s*"(https://creator\\.douyin\\.com[^"]+prefetch\\.json)"`)
);
if (mapped?.[1]) {
return decodeEscapedUrl(mapped[1]);
}
const discovered = Array.from(
new Set(
[...html.matchAll(/https:\/\/creator\.douyin\.com\/goofy\/douyin_creator_pc\/mono\/prefetch\/[^"]+prefetch\.json/g)].map(
(match) => decodeEscapedUrl(match[0])
)
)
);
return (
discovered.find((candidate) => candidate.includes(current.pathname.replace(/^\/creator-micro\//, ""))) ||
discovered[0] ||
`https://creator.douyin.com/goofy/douyin_creator_pc/mono/prefetch${current.pathname}/prefetch.json`
);
}
async function collectCreatorPrefetchResults(page) {
const prefetchUrl = await resolveCreatorPrefetchUrl(page);
return page.evaluate(async ({ prefetchUrl }) => {
try {
const prefetchResp = await fetch(prefetchUrl, { credentials: "same-origin" });
const prefetchText = await prefetchResp.text();
const prefetch = JSON.parse(prefetchText);
const results = [];
for (const api of prefetch?.apis || []) {
const target = new URL(api.url, window.location.origin);
for (const [key, value] of Object.entries(api.params || {})) {
target.searchParams.set(key, String(value));
}
const resp = await fetch(target.toString(), {
credentials: api.credentials || "same-origin",
});
const payload = await resp.json().catch(() => null);
results.push({
url: target.toString(),
payload,
});
}
return {
prefetch_url: prefetchUrl,
prefetch,
results,
};
} catch (error) {
return {
prefetch_url: prefetchUrl,
error: String(error),
results: [],
};
}
}, { prefetchUrl });
}
async function prepareProfilePage(page, options) {
await clickFirstVisible(page, [
"text=作品",
"text=视频",
"text=全部作品",
"[role='tab']:has-text('作品')"
]);
for (let index = 0; index < 3; index += 1) {
await clickFirstVisible(page, [
"text=展开",
"text=更多",
"text=查看全部"
]);
}
for (let index = 0; index < options.scrollCount; index += 1) {
await page.evaluate(() => window.scrollBy(0, window.innerHeight * 0.85));
await sleep(1200);
}
}
async function capturePageBundle(page, label, responseCapture, extra = {}) {
const html = await page.content();
const loginGateDetected =
html.includes("扫码登录") ||
html.includes("验证码登录") ||
html.includes("登录后免费畅享高清视频");
const antiBotDetected =
html.includes("window.byted_acrawler.init") ||
html.includes("__ac_signature") ||
html.includes("__ac_nonce");
const scripts = extractScriptPayloads(html);
const globals = await collectWindowGlobals(page);
const network = await responseCapture.stop();
const bundle = {
label,
captured_at: new Date().toISOString(),
page_url: page.url(),
page_title: await page.title().catch(() => ""),
page_meta: await page.evaluate(() => ({
href: window.location.href,
title: document.title,
text_excerpt: (document.body?.innerText || "").trim().slice(0, 8000)
})),
capture_hints: {
login_gate_detected: loginGateDetected,
anti_bot_detected: antiBotDetected
},
scripts,
globals,
network,
extra
};
return bundle;
}
async function saveJson(filePath, value) {
await ensureDir(path.dirname(filePath));
await fs.writeFile(filePath, JSON.stringify(value, null, 2), "utf8");
}
async function saveJsonSafe(filePath, value) {
try {
await saveJson(filePath, value);
} catch (error) {
console.error(`Failed to write ${filePath}: ${error?.message || error}`);
}
}
async function requestJson(urlString, { method = "GET", headers = {}, body = null, bodyPath = "" } = {}) {
const bodyMode = bodyPath ? "path" : body === null ? "none" : "text";
const bodyValue = bodyPath || (typeof body === "string" ? body : JSON.stringify(body));
const stdout = execFileSync(
"python3",
["-c", PYTHON_HTTP_BRIDGE, urlString, method, JSON.stringify(headers), bodyMode, bodyValue],
{ maxBuffer: 20 * 1024 * 1024, encoding: "utf8" }
);
const payload = JSON.parse(String(stdout || "").trim() || "{}");
if ((payload.status || 500) >= 400) {
throw new Error(`Request failed: ${payload.status} ${JSON.stringify(payload.data)}`);
}
return payload.data;
}
async function loginStoryForge(baseUrl, username, password) {
return requestJson(`${baseUrl.replace(/\/$/, "")}/v2/auth/login`, {
method: "POST",
headers: { "content-type": "application/json" },
body: { username, password }
});
}
async function syncCapture(baseUrl, token, bodyPath) {
return requestJson(`${baseUrl.replace(/\/$/, "")}/v2/douyin/accounts/sync`, {
method: "POST",
headers: {
"content-type": "application/json",
Authorization: `Bearer ${token}`
},
bodyPath
});
}
async function captureCreatorPages(context, options, runDir) {
const pages = [];
if (!options.creatorCenterEnabled) {
return pages;
}
for (const [index, url] of options.creatorCenterUrls.entries()) {
const page = await context.newPage();
const responseCapture = await createResponseCapture(page);
try {
console.error(`Capturing creator-center page: ${url}`);
await navigateAndSettle(page, url, options.waitMs);
const prefetchResults = await collectCreatorPrefetchResults(page);
const bundle = await capturePageBundle(page, "creator_center", responseCapture, {
creator_prefetch: prefetchResults
});
pages.push({
url: bundle.page_url,
title: bundle.page_title,
payload: bundle
});
await saveJson(
path.join(runDir, `creator-${String(index + 1).padStart(2, "0")}-${sanitizeName(bundle.page_title || bundle.page_url)}.json`),
bundle
);
} finally {
await page.close().catch(() => {});
}
}
return pages;
}
async function captureVideoPages(context, videoLinks, options, runDir) {
const pages = [];
for (const link of videoLinks.slice(0, Math.max(options.maxVideos, 0))) {
const page = await context.newPage();
const responseCapture = await createResponseCapture(page);
try {
console.error(`Capturing video page: ${link}`);
await navigateAndSettle(page, link, options.waitMs);
const bundle = await capturePageBundle(page, "video_detail", responseCapture, { source_link: link });
pages.push(bundle);
await saveJson(path.join(runDir, `video-${sanitizeName(link)}.json`), bundle);
} finally {
await page.close().catch(() => {});
}
}
return pages;
}
async function main() {
const options = parseArgs(process.argv.slice(2));
if (options.help) {
printHelp();
return;
}
if (!options.profileUrl) {
throw new Error("--profile-url is required");
}
if (
options.syncEnabled &&
!options.storyforgeToken &&
!(options.storyforgeUsername && options.storyforgePassword)
) {
throw new Error("Sync mode requires --storyforge-token or both --storyforge-username and --storyforge-password");
}
const runDir = path.join(
options.outputDir,
`${nowStamp()}-${sanitizeName(options.profileUrl.split("/").pop() || "douyin")}`
);
await ensureDir(runDir);
await ensureDir(options.stateDir);
const summary = {
profile_url: options.profileUrl,
output_dir: runDir,
video_link_count: 0,
captured_video_pages: 0,
captured_creator_pages: 0,
sync_enabled: options.syncEnabled,
status: "running"
};
await saveJsonSafe(path.join(runDir, "summary.json"), summary);
let storyforgeToken = options.storyforgeToken;
if (options.syncEnabled && !storyforgeToken) {
const auth = await loginStoryForge(
options.backendUrl,
options.storyforgeUsername,
options.storyforgePassword
);
storyforgeToken = auth.token;
await saveJson(path.join(runDir, "storyforge-login.json"), {
account: auth.account,
default_external_base_url: auth.default_external_base_url
});
}
const context = await chromium.launchPersistentContext(options.stateDir, {
headless: options.headless,
viewport: { width: 1440, height: 1024 },
args: ["--disable-blink-features=AutomationControlled"]
});
try {
const page = await context.newPage();
const responseCapture = await createResponseCapture(page);
console.error(`Opening profile page: ${options.profileUrl}`);
await navigateAndSettle(page, options.profileUrl, options.waitMs);
await maybePrompt(
`Browser opened ${options.profileUrl}.\nLog into Douyin if needed, solve any slider/captcha, and optionally click into the creator homepage before capture.`,
options.manualPrompt,
options.readyFile
);
await prepareProfilePage(page, options);
await sleep(options.waitMs);
const videoLinks = await collectVideoLinks(page);
console.error(`Collected ${videoLinks.length} candidate video links`);
const profileBundle = await capturePageBundle(page, "profile", responseCapture, { video_links: videoLinks });
await saveJson(path.join(runDir, "profile-bundle.json"), profileBundle);
await page.close().catch(() => {});
const creatorPages = await captureCreatorPages(context, options, runDir);
const videoPages = await captureVideoPages(context, videoLinks, options, runDir);
const syncBody = {
profile_url: options.profileUrl,
allow_creator_center_profile_fallback: options.allowCreatorCenterFallback,
compact_response: true,
manual_profile_payload: profileBundle,
manual_creator_pages: creatorPages,
manual_work_payloads: videoPages,
discovery_note: options.note || "browser-assisted capture"
};
const syncRequestPath = path.join(runDir, "storyforge-sync-request.json");
await saveJson(syncRequestPath, syncBody);
summary.video_link_count = videoLinks.length;
summary.captured_video_pages = videoPages.length;
summary.captured_creator_pages = creatorPages.length;
if (options.syncEnabled) {
const workspace = await syncCapture(options.backendUrl, storyforgeToken, syncRequestPath);
summary.sync_result = {
account_id: workspace.account?.id || "",
nickname: workspace.account?.nickname || "",
sync_errors: workspace.sync_errors || []
};
await saveJson(path.join(runDir, "storyforge-sync-response.json"), workspace);
}
summary.status = "completed";
await saveJson(path.join(runDir, "summary.json"), summary);
console.log(JSON.stringify(summary, null, 2));
} catch (error) {
summary.status = "failed";
summary.error = error?.stack || String(error);
await saveJsonSafe(path.join(runDir, "summary.json"), summary);
await saveJsonSafe(path.join(runDir, "storyforge-sync-error.json"), {
error: error?.stack || String(error)
});
throw error;
} finally {
await context.close().catch(() => {});
}
}
main().catch((error) => {
console.error(error?.stack || String(error));
process.exitCode = 1;
});

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,59 @@
{
"name": "storyforge-douyin-browser-capture",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "storyforge-douyin-browser-capture",
"version": "0.1.0",
"dependencies": {
"playwright": "^1.56.1"
}
},
"node_modules/fsevents": {
"version": "2.3.2",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
}
},
"node_modules/playwright": {
"version": "1.58.2",
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.58.2.tgz",
"integrity": "sha512-vA30H8Nvkq/cPBnNw4Q8TWz1EJyqgpuinBcHET0YVJVFldr8JDNiU9LaWAE1KqSkRYazuaBhTpB5ZzShOezQ6A==",
"license": "Apache-2.0",
"dependencies": {
"playwright-core": "1.58.2"
},
"bin": {
"playwright": "cli.js"
},
"engines": {
"node": ">=18"
},
"optionalDependencies": {
"fsevents": "2.3.2"
}
},
"node_modules/playwright-core": {
"version": "1.58.2",
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.58.2.tgz",
"integrity": "sha512-yZkEtftgwS8CsfYo7nm0KE8jsvm6i/PTgVtB8DL726wNf6H2IMsDuxCpJj59KDaxCtSnrWan2AeDqM7JBaultg==",
"license": "Apache-2.0",
"bin": {
"playwright-core": "cli.js"
},
"engines": {
"node": ">=18"
}
}
}
}

View File

@@ -0,0 +1,15 @@
{
"name": "storyforge-douyin-browser-capture",
"version": "0.1.0",
"private": true,
"type": "module",
"description": "Browser-assisted Douyin capture and sync tool for StoryForge",
"scripts": {
"capture": "node ./capture_and_sync.mjs",
"control-panel": "node ./control_panel.mjs",
"help": "node ./capture_and_sync.mjs --help"
},
"dependencies": {
"playwright": "^1.56.1"
}
}

58
scripts/smoke_business.sh Executable file
View File

@@ -0,0 +1,58 @@
#!/bin/sh
set -eu
BASE_URL="${STORYFORGE_BASE_URL:-http://127.0.0.1:8081}"
USERNAME="${STORYFORGE_USERNAME:-kris}"
PASSWORD="${STORYFORGE_PASSWORD:-Asd123456.}"
ACCOUNT_ID="${STORYFORGE_SMOKE_ACCOUNT_ID:-dyacct_c2b62842b228406cb48f05fac16fdfdf}"
python3 - <<'PY'
import json
import os
import urllib.request
base = os.environ.get("BASE_URL", "http://127.0.0.1:8081").rstrip("/")
username = os.environ.get("USERNAME", "kris")
password = os.environ.get("PASSWORD", "Asd123456.")
account_id = os.environ.get("ACCOUNT_ID", "dyacct_c2b62842b228406cb48f05fac16fdfdf")
login_req = urllib.request.Request(
base + "/v2/auth/login",
data=json.dumps({"username": username, "password": password}).encode(),
headers={"content-type": "application/json"},
)
with urllib.request.urlopen(login_req, timeout=20) as resp:
login = json.load(resp)
token = login["token"]
headers = {"authorization": "Bearer " + token}
checks = [
("/v2/douyin/accounts", "accounts"),
(f"/v2/douyin/accounts/{account_id}/workspace", "workspace"),
(f"/v2/douyin/accounts/{account_id}/videos?limit=5&sort_by=score", "videos"),
]
print("smoke login: ok")
for path, label in checks:
req = urllib.request.Request(base + path, headers=headers)
with urllib.request.urlopen(req, timeout=30) as resp:
payload = json.load(resp)
if label == "accounts":
summary = {"accounts": len(payload)}
elif label == "workspace":
summary = {
"account": payload.get("account", {}).get("nickname"),
"reports": len(payload.get("recent_reports") or []),
"linked_accounts": len(payload.get("linked_accounts") or []),
"high_score_threshold": (payload.get("video_workspace") or {}).get("high_score_threshold"),
}
else:
items = payload.get("items") or []
summary = {
"videos": len(items),
"first_title": items[0].get("title") if items else None,
"first_has_analysis": bool(items and items[0].get("latest_analysis")),
}
print(f"{label}: " + json.dumps(summary, ensure_ascii=False))
PY

39
scripts/start_business.sh Executable file
View File

@@ -0,0 +1,39 @@
#!/bin/sh
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
COMPOSE_FILE="$ROOT/docker-compose.yml"
cd "$ROOT"
docker compose -f "$COMPOSE_FILE" up -d --build collector n8n
python3 - <<'PY'
import time
import urllib.request
checks = [
("collector", "http://127.0.0.1:8081/healthz"),
("n8n", "http://127.0.0.1:5670/healthz"),
]
deadline = time.time() + 45
pending = dict(checks)
while pending and time.time() < deadline:
for name, url in list(pending.items()):
try:
with urllib.request.urlopen(url, timeout=5) as resp:
print(f"{name} ready: {resp.status}")
pending.pop(name, None)
except Exception:
pass
if pending:
time.sleep(1)
if pending:
print("startup timeout:", ", ".join(pending))
raise SystemExit(1)
PY
echo "business started"
echo "collector: http://127.0.0.1:8081/healthz"
echo "n8n: http://127.0.0.1:5670/healthz"

View File

@@ -1,25 +1,28 @@
#!/bin/sh
set -eu
ROOT="/Users/kris/code/StoryForge"
PID_FILE="$ROOT/data/collector/collector.pid"
LOG_FILE="$ROOT/data/collector/collector.log"
VENV="$ROOT/collector-service/.venv311"
mkdir -p "$ROOT/data/collector"
if [ ! -x "$VENV/bin/python" ]; then
/opt/homebrew/bin/python3.11 -m venv "$VENV"
. "$VENV/bin/activate"
pip install -q -r "$ROOT/collector-service/requirements.txt"
else
. "$VENV/bin/activate"
fi
if [ -f "$PID_FILE" ]; then
PID="$(cat "$PID_FILE" || true)"
if [ -n "$PID" ] && kill -0 "$PID" 2>/dev/null; then
echo "collector already running: $PID"
exit 0
fi
fi
cd "$ROOT/collector-service"
nohup "$VENV/bin/python" -m uvicorn app.main:app --host 0.0.0.0 --port 8081 >"$LOG_FILE" 2>&1 &
echo $! > "$PID_FILE"
echo "collector started: $(cat "$PID_FILE")"
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
COMPOSE_FILE="$ROOT/docker-compose.yml"
cd "$ROOT"
docker compose -f "$COMPOSE_FILE" up -d --build collector
python3 - <<'PY'
import time
import urllib.request
url = "http://127.0.0.1:8081/healthz"
deadline = time.time() + 30
last_error = ""
while time.time() < deadline:
try:
with urllib.request.urlopen(url, timeout=5) as resp:
print(f"collector ready: {resp.status} {resp.read().decode('utf-8', 'ignore')[:160]}")
raise SystemExit(0)
except Exception as exc:
last_error = str(exc)
time.sleep(1)
print(f"collector start timeout: {last_error}")
raise SystemExit(1)
PY

View File

@@ -0,0 +1,38 @@
#!/bin/sh
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
PORT="${DOUYIN_WORKBENCH_PORT:-3618}"
SCRIPT="$ROOT/scripts/douyin-browser-capture/control_panel.mjs"
LOG_FILE="${DOUYIN_WORKBENCH_LOG:-/tmp/storyforge-douyin-workbench.log}"
SESSION_NAME="${DOUYIN_WORKBENCH_SESSION:-storyforge-douyin-workbench}"
if lsof -nP -iTCP:"$PORT" -sTCP:LISTEN >/dev/null 2>&1; then
echo "douyin workbench already running: http://127.0.0.1:$PORT/workbench"
exit 0
fi
screen -wipe >/dev/null 2>&1 || true
screen -S "$SESSION_NAME" -X quit >/dev/null 2>&1 || true
screen -dmS "$SESSION_NAME" /bin/sh -lc "exec env PORT='$PORT' node '$SCRIPT' >>'$LOG_FILE' 2>&1"
python3 - <<'PY'
import os
import time
import urllib.request
port = os.environ.get("PORT", "3618")
url = f"http://127.0.0.1:{port}/workbench"
deadline = time.time() + 15
last_error = ""
while time.time() < deadline:
try:
with urllib.request.urlopen(url, timeout=3) as resp:
print(f"douyin workbench ready: {resp.status} {url}")
raise SystemExit(0)
except Exception as exc:
last_error = str(exc)
time.sleep(0.5)
print(f"douyin workbench start timeout: {last_error}")
raise SystemExit(1)
PY

22
scripts/status_business.sh Executable file
View File

@@ -0,0 +1,22 @@
#!/bin/sh
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
COMPOSE_FILE="$ROOT/docker-compose.yml"
cd "$ROOT"
docker compose -f "$COMPOSE_FILE" ps
echo "---"
python3 - <<'PY'
import urllib.request
for name, url in [
("collector", "http://127.0.0.1:8081/healthz"),
("n8n", "http://127.0.0.1:5670/healthz"),
]:
try:
with urllib.request.urlopen(url, timeout=5) as resp:
print(f"{name}: {resp.status} {resp.read().decode('utf-8', 'ignore')[:200]}")
except Exception as exc:
print(f"{name}: ERROR {exc}")
PY

View File

@@ -1,15 +1,21 @@
#!/bin/sh
set -eu
PID_FILE="/Users/kris/code/StoryForge/data/collector/collector.pid"
if [ -f "$PID_FILE" ]; then
PID="$(cat "$PID_FILE" || true)"
if [ -n "$PID" ] && kill -0 "$PID" 2>/dev/null; then
echo "running:$PID"
exit 0
fi
fi
if lsof -nP -iTCP:8081 -sTCP:LISTEN >/dev/null 2>&1; then
echo "running:port"
else
echo "stopped"
fi
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
COMPOSE_FILE="$ROOT/docker-compose.yml"
cd "$ROOT"
docker compose -f "$COMPOSE_FILE" ps collector
echo "---"
python3 - <<'PY'
import urllib.request
url = "http://127.0.0.1:8081/healthz"
try:
with urllib.request.urlopen(url, timeout=5) as resp:
print(f"collector health: {resp.status}")
print(resp.read().decode("utf-8", "ignore")[:400])
except Exception as exc:
print(f"collector health error: {exc}")
raise SystemExit(1)
PY

View File

@@ -0,0 +1,23 @@
#!/bin/sh
set -eu
PORT="${DOUYIN_WORKBENCH_PORT:-3618}"
SESSION_NAME="${DOUYIN_WORKBENCH_SESSION:-storyforge-douyin-workbench}"
if ! lsof -nP -iTCP:"$PORT" -sTCP:LISTEN >/dev/null 2>&1; then
echo "douyin workbench stopped"
exit 1
fi
screen -ls | grep "$SESSION_NAME" || true
echo "---"
python3 - <<'PY'
import os
import urllib.request
port = os.environ.get("PORT", "3618")
for path in ("/workbench", "/"):
url = f"http://127.0.0.1:{port}{path}"
with urllib.request.urlopen(url, timeout=5) as resp:
print(f"{path}: {resp.status}")
PY

View File

@@ -1,15 +1,9 @@
#!/bin/sh
set -eu
PID_FILE="/Users/kris/code/StoryForge/data/collector/collector.pid"
if [ ! -f "$PID_FILE" ]; then
echo "collector not running"
exit 0
fi
PID="$(cat "$PID_FILE" || true)"
if [ -n "$PID" ] && kill -0 "$PID" 2>/dev/null; then
kill "$PID"
echo "collector stopped: $PID"
else
echo "collector pid stale: $PID"
fi
rm -f "$PID_FILE"
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
COMPOSE_FILE="$ROOT/docker-compose.yml"
cd "$ROOT"
docker compose -f "$COMPOSE_FILE" stop collector
echo "collector stopped"

View File

@@ -0,0 +1,9 @@
#!/bin/sh
set -eu
PORT="${DOUYIN_WORKBENCH_PORT:-3618}"
SESSION_NAME="${DOUYIN_WORKBENCH_SESSION:-storyforge-douyin-workbench}"
screen -S "$SESSION_NAME" -X quit >/dev/null 2>&1 || true
lsof -tiTCP:"$PORT" -sTCP:LISTEN | xargs -r kill
echo "douyin workbench stopped: $PORT"