重写为 AI 情报 Agent 并接入 Jenkins 流水线
This commit is contained in:
@@ -0,0 +1,20 @@
|
|||||||
|
SIGNALSCOUT_APP_NAME=SignalScout
|
||||||
|
SIGNALSCOUT_ENV=dev
|
||||||
|
SIGNALSCOUT_DATABASE_URL=mysql+pymysql://signalscout:signalscout@127.0.0.1:3306/signalscout?charset=utf8mb4
|
||||||
|
SIGNALSCOUT_LLM_BASE_URL=https://api.zayuapi.com/v1
|
||||||
|
SIGNALSCOUT_LLM_API_KEY=
|
||||||
|
SIGNALSCOUT_LLM_MODEL=grok-4.3
|
||||||
|
SIGNALSCOUT_LLM_TIMEOUT_SECONDS=90
|
||||||
|
SIGNALSCOUT_LLM_MAX_TOKENS=4000
|
||||||
|
SIGNALSCOUT_NEWS_RECENT_DAYS=3
|
||||||
|
SIGNALSCOUT_NEWS_MAX_ITEMS=12
|
||||||
|
SIGNALSCOUT_NEWS_SEARCH_MAX_RECORDS=20
|
||||||
|
SIGNALSCOUT_GITHUB_RECENT_DAYS=30
|
||||||
|
SIGNALSCOUT_GITHUB_MIN_STARS=20
|
||||||
|
SIGNALSCOUT_GITHUB_MAX_PROJECTS=5
|
||||||
|
SIGNALSCOUT_AGENT_TIMEZONE=Asia/Shanghai
|
||||||
|
SIGNALSCOUT_AGENT_CRON_HOUR=8
|
||||||
|
SIGNALSCOUT_AGENT_CRON_MINUTE=30
|
||||||
|
SIGNALSCOUT_REPORT_DIR=reports
|
||||||
|
SIGNALSCOUT_LOG_FILE=logs/app.log
|
||||||
|
SIGNALSCOUT_LOG_TAIL_LINES=200
|
||||||
+10
@@ -0,0 +1,10 @@
|
|||||||
|
.env
|
||||||
|
.venv/
|
||||||
|
__pycache__/
|
||||||
|
.pytest_cache/
|
||||||
|
.ruff_cache/
|
||||||
|
*.pyc
|
||||||
|
*.egg-info/
|
||||||
|
reports/
|
||||||
|
logs/
|
||||||
|
data/
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Design
|
||||||
|
|
||||||
|
## Theme
|
||||||
|
|
||||||
|
An operator reviews AI intelligence on a desktop monitor in a normal evening work setting, scanning sources and deciding what to read next. The UI should be light, quiet, and information-forward.
|
||||||
|
|
||||||
|
## Color
|
||||||
|
|
||||||
|
- Use OKLCH color values.
|
||||||
|
- Background: softly tinted neutral, not pure white.
|
||||||
|
- Surface: near-white neutral panels with subtle borders.
|
||||||
|
- Text: ink-like neutral with a small cool tint.
|
||||||
|
- Accent: restrained green-cyan for primary actions, active navigation, and selected states.
|
||||||
|
- Semantic states: red for failed runs, amber for warnings, green for completed runs.
|
||||||
|
|
||||||
|
## Typography
|
||||||
|
|
||||||
|
- Use system UI fonts: `-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif`.
|
||||||
|
- Keep a compact product scale: strong page title, restrained section headings, dense row text.
|
||||||
|
- Body copy should stay readable for Chinese and English mixed content.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
- App shell with a narrow left rail and a content workspace.
|
||||||
|
- First screen prioritizes collected signals and latest report, not abstract metrics.
|
||||||
|
- Use tables/lists for intelligence items; use cards only for repeated signal rows and summary panels where framing helps.
|
||||||
|
- Mobile collapses to a single-column review flow.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
- Primary button for triggering a collection run.
|
||||||
|
- Status pills for run state.
|
||||||
|
- Filter/search controls for topics and entities.
|
||||||
|
- Signal rows with title, summary, source, topic, entities, and published time.
|
||||||
|
- Report reader panel with latest Markdown content.
|
||||||
|
- Empty state that says no collected data exists yet and offers a run action.
|
||||||
|
|
||||||
|
## Motion
|
||||||
|
|
||||||
|
Use short 150-200ms transitions for hover, selection, and loading states. Respect reduced motion.
|
||||||
Vendored
+28
@@ -0,0 +1,28 @@
|
|||||||
|
pipeline {
|
||||||
|
agent any
|
||||||
|
|
||||||
|
environment {
|
||||||
|
REMOTE_HOST = credentials('agent-desk-ssh-host')
|
||||||
|
REMOTE_USER = credentials('agent-desk-ssh-user')
|
||||||
|
SSH_CREDENTIALS_ID = 'agent-desk-server-ssh'
|
||||||
|
REPO_URL = 'http://gitadmin:12345678911@gitea:3000/gitadmin/signalscout-backend.git'
|
||||||
|
PUBLIC_READY_URL = 'https://signalscout.jaycode.online/health'
|
||||||
|
}
|
||||||
|
|
||||||
|
stages {
|
||||||
|
stage('Deploy from Gitea') {
|
||||||
|
steps {
|
||||||
|
withCredentials([sshUserPrivateKey(credentialsId: "${SSH_CREDENTIALS_ID}", keyFileVariable: 'SSH_KEY')]) {
|
||||||
|
// Jenkins只负责触发服务器部署;依赖安装、迁移、测试和服务重启都由目标服务器脚本统一完成。
|
||||||
|
sh '''
|
||||||
|
chmod 600 "${SSH_KEY}"
|
||||||
|
ssh -o StrictHostKeyChecking=no "${REMOTE_USER}@${REMOTE_HOST}" \
|
||||||
|
-i "${SSH_KEY}" \
|
||||||
|
"REPO_URL='${REPO_URL}' PUBLIC_READY_URL='${PUBLIC_READY_URL}' bash -s" \
|
||||||
|
< deployment/jenkins/deploy_backend.sh
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
.PHONY: install lint test migrate seed run
|
||||||
|
|
||||||
|
PYTHON := /mnt/d/Python/Python39/python.exe
|
||||||
|
PYTHON_NO_CACHE := $(PYTHON) -B
|
||||||
|
|
||||||
|
install:
|
||||||
|
$(PYTHON) -m pip install -e ".[dev]"
|
||||||
|
|
||||||
|
lint:
|
||||||
|
$(PYTHON_NO_CACHE) -m ruff check .
|
||||||
|
|
||||||
|
test:
|
||||||
|
$(PYTHON_NO_CACHE) -m pytest
|
||||||
|
|
||||||
|
migrate:
|
||||||
|
$(PYTHON_NO_CACHE) -m alembic upgrade head
|
||||||
|
|
||||||
|
seed:
|
||||||
|
$(PYTHON_NO_CACHE) scripts/seed_topics.py
|
||||||
|
|
||||||
|
run:
|
||||||
|
$(PYTHON_NO_CACHE) -m uvicorn app.main:app --host 127.0.0.1 --port 8010
|
||||||
+33
@@ -0,0 +1,33 @@
|
|||||||
|
# Product
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
product
|
||||||
|
|
||||||
|
## Users
|
||||||
|
|
||||||
|
SignalScout is used by one operator who wants a daily AI-generated intelligence brief without manually searching across news, X, GitHub, and company announcements. The user is usually checking what changed, which projects matter, and which links deserve follow-up.
|
||||||
|
|
||||||
|
## Product Purpose
|
||||||
|
|
||||||
|
The product searches dynamic news candidates and recent high-heat GitHub project candidates, then uses Grok to structure AI news, model releases, agent framework movement, product launches, and funding signals. Success means the user can open one dashboard, see concrete collected items with sources and context, inspect the latest daily report, and trigger a fresh collection run when needed.
|
||||||
|
|
||||||
|
## Brand Personality
|
||||||
|
|
||||||
|
Quiet, sharp, operational. The interface should feel like an intelligence desk rather than a marketing site.
|
||||||
|
|
||||||
|
## Anti-references
|
||||||
|
|
||||||
|
Do not make a landing page, generic SaaS hero, decorative card grid, or empty analytics dashboard. Avoid fake metrics that do not come from collected data. Avoid loud neon AI styling, glass panels, gradient text, and ornamental charts.
|
||||||
|
|
||||||
|
## Design Principles
|
||||||
|
|
||||||
|
- Evidence first: every important item should expose its source and enough context to verify it.
|
||||||
|
- Workbench over brochure: the first screen should help the user read, filter, and act.
|
||||||
|
- Dense but calm: support repeated daily review without visual noise.
|
||||||
|
- Agent output is inspectable: show summaries, entities, run state, and report text clearly.
|
||||||
|
- One path: the UI reflects the current News Search + GitHub + Grok + MySQL pipeline.
|
||||||
|
|
||||||
|
## Accessibility & Inclusion
|
||||||
|
|
||||||
|
Use semantic controls, visible focus states, sufficient contrast, reduced-motion friendly transitions, and readable Chinese/English mixed content. Target practical WCAG AA behavior for this private dashboard.
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# SignalScout
|
||||||
|
|
||||||
|
SignalScout 是一个定时运行的 AI 情报 Agent。它先用动态新闻搜索和 GitHub 官方搜索抓取候选证据,再使用 Grok 通过中转 API 生成最新 AI 新闻、模型发布、Agent 工具、融资动态和 GitHub 热门项目日报,并把结构化情报保存到 MySQL。
|
||||||
|
|
||||||
|
## 主流程
|
||||||
|
|
||||||
|
```text
|
||||||
|
Scheduler/API
|
||||||
|
-> AgentRunner
|
||||||
|
-> News Search API 采集近期 AI 新闻候选
|
||||||
|
-> GitHub Search API 采集近期热门 AI 项目候选
|
||||||
|
-> Grok 研判新闻候选与 GitHub 候选
|
||||||
|
-> MySQL signals/reports
|
||||||
|
-> Markdown report file
|
||||||
|
```
|
||||||
|
|
||||||
|
## 本地配置
|
||||||
|
|
||||||
|
复制环境变量:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
```
|
||||||
|
|
||||||
|
关键配置:
|
||||||
|
|
||||||
|
```env
|
||||||
|
SIGNALSCOUT_DATABASE_URL=mysql+pymysql://user:password@host:3306/signalscout?charset=utf8mb4
|
||||||
|
SIGNALSCOUT_LLM_BASE_URL=https://api.zayuapi.com/v1
|
||||||
|
SIGNALSCOUT_LLM_API_KEY=你的中转站 key
|
||||||
|
SIGNALSCOUT_LLM_MODEL=grok-4.3
|
||||||
|
SIGNALSCOUT_NEWS_RECENT_DAYS=3
|
||||||
|
SIGNALSCOUT_NEWS_MAX_ITEMS=12
|
||||||
|
SIGNALSCOUT_NEWS_SEARCH_MAX_RECORDS=20
|
||||||
|
SIGNALSCOUT_GITHUB_RECENT_DAYS=30
|
||||||
|
SIGNALSCOUT_GITHUB_MIN_STARS=20
|
||||||
|
SIGNALSCOUT_GITHUB_MAX_PROJECTS=5
|
||||||
|
```
|
||||||
|
|
||||||
|
## 本机运行
|
||||||
|
|
||||||
|
按项目规则,本机 Python 使用:
|
||||||
|
|
||||||
|
```text
|
||||||
|
D:\Python\Python39\python.exe
|
||||||
|
```
|
||||||
|
|
||||||
|
在 WSL shell 中对应命令已经写进 `Makefile`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make install
|
||||||
|
make migrate
|
||||||
|
make seed
|
||||||
|
make run
|
||||||
|
```
|
||||||
|
|
||||||
|
后端启动后访问:
|
||||||
|
|
||||||
|
```text
|
||||||
|
http://127.0.0.1:8010/health
|
||||||
|
http://127.0.0.1:8010/docs
|
||||||
|
```
|
||||||
|
|
||||||
|
## API
|
||||||
|
|
||||||
|
- `GET /health`:服务健康检查
|
||||||
|
- `POST /agent/runs/daily`:手动触发每日 Agent
|
||||||
|
- `POST /agent/runs/daily/stream`:以 SSE 方式触发每日 Agent
|
||||||
|
- `GET /agent/runs`:查看最近运行记录
|
||||||
|
- `GET /signals`:查看结构化情报
|
||||||
|
- `GET /reports/latest`:查看最新日报
|
||||||
|
- `GET /dashboard`:获取仪表盘数据
|
||||||
|
- `POST /topics`:新增 Agent 跟踪主题
|
||||||
|
- `GET /topics`:查看 Agent 跟踪主题
|
||||||
|
- `GET /logs/recent`:查看最近运行日志
|
||||||
|
|
||||||
|
## 部署
|
||||||
|
|
||||||
|
部署文件使用和 `agent-desk` 一致的 Gitea + Jenkins 流水线结构:
|
||||||
|
|
||||||
|
- Gitea 仓库:`https://gitea.jaycode.online/gitadmin/signalscout-backend`
|
||||||
|
- Jenkins Job:`signalscout-backend`
|
||||||
|
- 源码目录:`/opt/signalscout-source`
|
||||||
|
- 发布目录:`/opt/signalscout`
|
||||||
|
- 部署脚本:`deployment/jenkins/deploy_backend.sh`
|
||||||
|
- systemd 服务:`deployment/systemd/signalscout.service`
|
||||||
|
- Jenkins 流水线:`Jenkinsfile`
|
||||||
|
|
||||||
|
默认域名:
|
||||||
|
|
||||||
|
```text
|
||||||
|
https://signalscout.jaycode.online/
|
||||||
|
https://gitea.jaycode.online/
|
||||||
|
https://jenkins.jaycode.online/
|
||||||
|
```
|
||||||
+37
@@ -0,0 +1,37 @@
|
|||||||
|
[alembic]
|
||||||
|
script_location = migrations
|
||||||
|
prepend_sys_path = .
|
||||||
|
sqlalchemy.url = mysql+pymysql://signalscout:signalscout@127.0.0.1:3306/signalscout?charset=utf8mb4
|
||||||
|
|
||||||
|
[loggers]
|
||||||
|
keys = root,sqlalchemy,alembic
|
||||||
|
|
||||||
|
[handlers]
|
||||||
|
keys = console
|
||||||
|
|
||||||
|
[formatters]
|
||||||
|
keys = generic
|
||||||
|
|
||||||
|
[logger_root]
|
||||||
|
level = WARN
|
||||||
|
handlers = console
|
||||||
|
qualname =
|
||||||
|
|
||||||
|
[logger_sqlalchemy]
|
||||||
|
level = WARN
|
||||||
|
handlers =
|
||||||
|
qualname = sqlalchemy.engine
|
||||||
|
|
||||||
|
[logger_alembic]
|
||||||
|
level = INFO
|
||||||
|
handlers =
|
||||||
|
qualname = alembic
|
||||||
|
|
||||||
|
[handler_console]
|
||||||
|
class = StreamHandler
|
||||||
|
args = (sys.stderr,)
|
||||||
|
level = NOTSET
|
||||||
|
formatter = generic
|
||||||
|
|
||||||
|
[formatter_generic]
|
||||||
|
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""SignalScout application package."""
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Agent orchestration package."""
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
from datetime import date
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
class ReportWriter:
|
||||||
|
"""把人可读日报写入本地文件。"""
|
||||||
|
|
||||||
|
def __init__(self, base_dir: Path) -> None:
|
||||||
|
self.base_dir = base_dir
|
||||||
|
|
||||||
|
def write_daily_report(self, run_date: date, content_md: str) -> Path:
|
||||||
|
# 日报按日期落盘,便于人工浏览和备份。
|
||||||
|
self.base_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
report_path = self.base_dir / f"{run_date.isoformat()}.md"
|
||||||
|
report_path.write_text(content_md, encoding="utf-8")
|
||||||
|
return report_path
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from datetime import date
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.agent.report_writer import ReportWriter
|
||||||
|
from app.core.config import Settings
|
||||||
|
from app.db.repository import AgentRepository
|
||||||
|
from app.integrations.github_client import GitHubHotProjectClient
|
||||||
|
from app.integrations.news_search_client import NewsSearchClient
|
||||||
|
from app.llm.grok_client import GrokIntelligenceClient
|
||||||
|
from app.schemas.agent import RunResponse
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class AgentRunner:
|
||||||
|
"""Coordinates source collection, persistence, and report writing."""
|
||||||
|
|
||||||
|
def __init__(self, db: Session, settings: Settings) -> None:
|
||||||
|
self.repo = AgentRepository(db)
|
||||||
|
self.db = db
|
||||||
|
self.github = GitHubHotProjectClient(settings)
|
||||||
|
self.news = NewsSearchClient(settings)
|
||||||
|
self.grok = GrokIntelligenceClient(settings)
|
||||||
|
self.writer = ReportWriter(settings.report_dir)
|
||||||
|
|
||||||
|
def run_daily(self, run_date: Optional[date] = None) -> RunResponse:
|
||||||
|
# Runner统一编排候选采集、模型研判和持久化,保证一次运行只有一种业务路径。
|
||||||
|
current_date = run_date or date.today()
|
||||||
|
request_payload = {
|
||||||
|
"run_date": current_date.isoformat(),
|
||||||
|
"mode": "grok_agent",
|
||||||
|
}
|
||||||
|
run = self.repo.create_run(request_payload=request_payload)
|
||||||
|
logger.info("agent_run_started run_id=%s run_date=%s mode=grok_agent", run.id, current_date)
|
||||||
|
|
||||||
|
try:
|
||||||
|
topics = self.repo.list_enabled_topics()
|
||||||
|
news_candidates = self.news.collect_ai_news(run_date=current_date)
|
||||||
|
github_projects = self.github.collect_hot_projects(run_date=current_date)
|
||||||
|
result = self.grok.collect_daily_intelligence(
|
||||||
|
topics=topics,
|
||||||
|
news_candidates=news_candidates,
|
||||||
|
github_projects=github_projects,
|
||||||
|
run_date=current_date,
|
||||||
|
)
|
||||||
|
signals = self.repo.save_signals(run=run, items=result.signals)
|
||||||
|
report_path = self.writer.write_daily_report(current_date, result.report_markdown)
|
||||||
|
self.repo.save_report(
|
||||||
|
run=run,
|
||||||
|
title=result.title,
|
||||||
|
content_md=result.report_markdown,
|
||||||
|
file_path=str(report_path),
|
||||||
|
)
|
||||||
|
self.repo.mark_run_complete(
|
||||||
|
run=run,
|
||||||
|
raw_response={
|
||||||
|
"mode": "grok_agent",
|
||||||
|
"topic_count": len(topics),
|
||||||
|
"news_candidate_count": len(news_candidates),
|
||||||
|
"github_project_count": len(github_projects),
|
||||||
|
"model": self.grok.settings.llm_model,
|
||||||
|
"model_response": result.raw_response,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
self.db.commit()
|
||||||
|
logger.info(
|
||||||
|
"agent_run_completed run_id=%s signals=%s report_path=%s",
|
||||||
|
run.id,
|
||||||
|
len(signals),
|
||||||
|
report_path,
|
||||||
|
)
|
||||||
|
return RunResponse(run_id=run.id, status=run.status, report_path=str(report_path))
|
||||||
|
except Exception as exc:
|
||||||
|
self.repo.mark_run_failed(run=run, error=str(exc))
|
||||||
|
self.db.commit()
|
||||||
|
logger.exception("agent_run_failed run_id=%s error=%s", run.id, exc)
|
||||||
|
raise
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
from apscheduler.schedulers.background import BackgroundScheduler
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.agent.runner import AgentRunner
|
||||||
|
from app.core.config import Settings
|
||||||
|
from app.db.session import SessionLocal
|
||||||
|
|
||||||
|
|
||||||
|
def run_scheduled_job(settings: Settings) -> None:
|
||||||
|
# 定时任务不在请求生命周期内,需要自己创建和关闭数据库会话。
|
||||||
|
db: Session = SessionLocal()
|
||||||
|
try:
|
||||||
|
AgentRunner(db=db, settings=settings).run_daily()
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def build_scheduler(settings: Settings) -> BackgroundScheduler:
|
||||||
|
# APScheduler负责每日后台运行,服务启动后自动挂载任务。
|
||||||
|
scheduler = BackgroundScheduler(timezone=settings.agent_timezone)
|
||||||
|
scheduler.add_job(
|
||||||
|
run_scheduled_job,
|
||||||
|
trigger="cron",
|
||||||
|
hour=settings.agent_cron_hour,
|
||||||
|
minute=settings.agent_cron_minute,
|
||||||
|
args=[settings],
|
||||||
|
id="daily-intelligence",
|
||||||
|
replace_existing=True,
|
||||||
|
)
|
||||||
|
return scheduler
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from datetime import date
|
||||||
|
from typing import Iterator, Optional
|
||||||
|
|
||||||
|
from app.agent.report_writer import ReportWriter
|
||||||
|
from app.core.config import Settings
|
||||||
|
from app.db.models import AgentRun, Signal
|
||||||
|
from app.db.repository import AgentRepository
|
||||||
|
from app.db.session import SessionLocal
|
||||||
|
from app.integrations.github_client import GitHubHotProjectClient
|
||||||
|
from app.integrations.news_search_client import NewsSearchClient
|
||||||
|
from app.llm.grok_client import GrokIntelligenceClient
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def stream_daily_agent_events(settings: Settings, run_date: Optional[date] = None) -> Iterator[str]:
|
||||||
|
"""以SSE方式运行每日Agent,并在模型完成后逐条发送结构化情报。"""
|
||||||
|
db = SessionLocal()
|
||||||
|
repo = AgentRepository(db)
|
||||||
|
current_date = run_date or date.today()
|
||||||
|
run: Optional[AgentRun] = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
request_payload = {
|
||||||
|
"run_date": current_date.isoformat(),
|
||||||
|
"mode": "grok_agent_stream",
|
||||||
|
}
|
||||||
|
run = repo.create_run(request_payload=request_payload)
|
||||||
|
db.commit()
|
||||||
|
logger.info("agent_stream_started run_id=%s run_date=%s mode=grok_agent", run.id, current_date)
|
||||||
|
yield _sse("started", {"run_id": run.id, "status": run.status})
|
||||||
|
|
||||||
|
news = NewsSearchClient(settings)
|
||||||
|
github = GitHubHotProjectClient(settings)
|
||||||
|
grok = GrokIntelligenceClient(settings)
|
||||||
|
topics = repo.list_enabled_topics()
|
||||||
|
yield _sse("stage", {"run_id": run.id, "stage": "news"})
|
||||||
|
news_candidates = news.collect_ai_news(run_date=current_date)
|
||||||
|
yield _sse("stage", {"run_id": run.id, "stage": "github"})
|
||||||
|
github_projects = github.collect_hot_projects(run_date=current_date)
|
||||||
|
yield _sse("stage", {"run_id": run.id, "stage": "grok"})
|
||||||
|
result = grok.collect_daily_intelligence(
|
||||||
|
topics=topics,
|
||||||
|
news_candidates=news_candidates,
|
||||||
|
github_projects=github_projects,
|
||||||
|
run_date=current_date,
|
||||||
|
)
|
||||||
|
|
||||||
|
saved_count = 0
|
||||||
|
final_signals = []
|
||||||
|
for item in result.signals:
|
||||||
|
# 模型一次性完成筛选和摘要,SSE只负责把最终情报逐条落库并推给前端。
|
||||||
|
signal = repo.save_or_update_signal(run=run, item=item)
|
||||||
|
if signal is None:
|
||||||
|
continue
|
||||||
|
db.commit()
|
||||||
|
db.refresh(signal)
|
||||||
|
final_signals.append(signal)
|
||||||
|
saved_count += 1
|
||||||
|
yield _sse("candidate", {"signal": _signal_payload(signal), "saved_count": saved_count})
|
||||||
|
|
||||||
|
writer = ReportWriter(settings.report_dir)
|
||||||
|
report_path = writer.write_daily_report(current_date, result.report_markdown)
|
||||||
|
repo.save_report(
|
||||||
|
run=run,
|
||||||
|
title=result.title,
|
||||||
|
content_md=result.report_markdown,
|
||||||
|
file_path=str(report_path),
|
||||||
|
)
|
||||||
|
repo.mark_run_complete(
|
||||||
|
run=run,
|
||||||
|
raw_response={
|
||||||
|
"mode": "grok_agent_stream",
|
||||||
|
"topic_count": len(topics),
|
||||||
|
"news_candidate_count": len(news_candidates),
|
||||||
|
"github_project_count": len(github_projects),
|
||||||
|
"model": settings.llm_model,
|
||||||
|
"model_response": result.raw_response,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
logger.info(
|
||||||
|
"agent_stream_completed run_id=%s candidates=%s final_signals=%s report_path=%s",
|
||||||
|
run.id,
|
||||||
|
saved_count,
|
||||||
|
len(final_signals),
|
||||||
|
report_path,
|
||||||
|
)
|
||||||
|
yield _sse(
|
||||||
|
"completed",
|
||||||
|
{
|
||||||
|
"run_id": run.id,
|
||||||
|
"status": "completed",
|
||||||
|
"candidate_count": saved_count,
|
||||||
|
"final_signal_count": len(final_signals),
|
||||||
|
"report_path": str(report_path),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
if run is not None:
|
||||||
|
repo.mark_run_failed(run=run, error=str(exc))
|
||||||
|
db.commit()
|
||||||
|
logger.exception("agent_stream_failed run_id=%s error=%s", run.id, exc)
|
||||||
|
yield _sse("failed", {"run_id": run.id, "status": "failed", "error": str(exc)})
|
||||||
|
else:
|
||||||
|
logger.exception("agent_stream_failed_before_run error=%s", exc)
|
||||||
|
yield _sse("failed", {"run_id": None, "status": "failed", "error": str(exc)})
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _signal_payload(signal: Signal) -> dict:
|
||||||
|
# SSE载荷只使用JSON安全字段,方便前端直接追加到列表。
|
||||||
|
return {
|
||||||
|
"id": signal.id,
|
||||||
|
"run_id": signal.run_id,
|
||||||
|
"topic": signal.topic,
|
||||||
|
"title": signal.title,
|
||||||
|
"summary": signal.summary,
|
||||||
|
"source_url": signal.source_url,
|
||||||
|
"source_name": signal.source_name,
|
||||||
|
"published_at": signal.published_at.isoformat() if signal.published_at else None,
|
||||||
|
"importance": signal.importance,
|
||||||
|
"entities": signal.entities,
|
||||||
|
"created_at": signal.created_at.isoformat() if signal.created_at else None,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _sse(event: str, data: dict) -> str:
|
||||||
|
# 标准SSE帧让前端可以通过流式请求消费事件。
|
||||||
|
payload = json.dumps(data, ensure_ascii=False)
|
||||||
|
return f"event: {event}\ndata: {payload}\n\n"
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""HTTP API routes."""
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import date
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends
|
||||||
|
from fastapi.responses import StreamingResponse
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.agent.runner import AgentRunner
|
||||||
|
from app.agent.streaming_runner import stream_daily_agent_events
|
||||||
|
from app.core.config import Settings, get_settings
|
||||||
|
from app.db.models import AgentRun, Report, Signal, Topic
|
||||||
|
from app.db.session import get_db
|
||||||
|
from app.schemas.agent import (
|
||||||
|
DashboardRead,
|
||||||
|
ReportRead,
|
||||||
|
RunRead,
|
||||||
|
RunResponse,
|
||||||
|
SignalRead,
|
||||||
|
TopicCreate,
|
||||||
|
TopicRead,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/health")
|
||||||
|
def health(settings: Settings = Depends(get_settings)) -> dict:
|
||||||
|
# 健康检查只返回服务身份,避免暴露密钥和内部地址。
|
||||||
|
return {"status": "ok", "app": settings.app_name, "env": settings.env}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/agent/runs/daily", response_model=RunResponse)
|
||||||
|
def run_daily_agent(
|
||||||
|
run_date: Optional[date] = None,
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
settings: Settings = Depends(get_settings),
|
||||||
|
) -> RunResponse:
|
||||||
|
# 手动触发和定时任务共用同一个Runner,保证行为一致。
|
||||||
|
return AgentRunner(db=db, settings=settings).run_daily(run_date=run_date)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/agent/runs/daily/stream")
|
||||||
|
def stream_daily_agent(
|
||||||
|
run_date: Optional[date] = None,
|
||||||
|
settings: Settings = Depends(get_settings),
|
||||||
|
) -> StreamingResponse:
|
||||||
|
# 流式接口立即启动Agent,并把最终结构化情报逐条推送给前端。
|
||||||
|
return StreamingResponse(
|
||||||
|
stream_daily_agent_events(settings=settings, run_date=run_date),
|
||||||
|
media_type="text/event-stream",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/agent/runs", response_model=list[RunRead])
|
||||||
|
def list_runs(db: Session = Depends(get_db)) -> list[AgentRun]:
|
||||||
|
# 运行列表保持紧凑,方便仪表盘和命令行查看最近状态。
|
||||||
|
return list(db.scalars(select(AgentRun).order_by(AgentRun.id.desc()).limit(20)).all())
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/signals", response_model=list[SignalRead])
|
||||||
|
def list_signals(limit: int = 50, db: Session = Depends(get_db)) -> list[Signal]:
|
||||||
|
# 情报列表按采集时间倒序返回,是仪表盘的主要阅读入口。
|
||||||
|
safe_limit = max(1, min(limit, 200))
|
||||||
|
return list(
|
||||||
|
db.scalars(_visible_signals_query().order_by(Signal.id.desc()).limit(safe_limit)).all()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/reports/latest", response_model=Optional[ReportRead])
|
||||||
|
def latest_report(db: Session = Depends(get_db)) -> Optional[Report]:
|
||||||
|
# 最新日报提供一份可直接阅读的每日简报。
|
||||||
|
return db.scalar(select(Report).order_by(Report.id.desc()).limit(1))
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/dashboard", response_model=DashboardRead)
|
||||||
|
def dashboard(db: Session = Depends(get_db)) -> DashboardRead:
|
||||||
|
# 单个仪表盘接口返回运行、信号、日报和主题,前端不需要理解内部调度细节。
|
||||||
|
runs = list(db.scalars(select(AgentRun).order_by(AgentRun.id.desc()).limit(10)).all())
|
||||||
|
signals = list(db.scalars(_visible_signals_query().order_by(Signal.id.desc()).limit(80)).all())
|
||||||
|
report = db.scalar(select(Report).order_by(Report.id.desc()).limit(1))
|
||||||
|
topics = list(db.scalars(select(Topic).order_by(Topic.id.asc())).all())
|
||||||
|
return DashboardRead(
|
||||||
|
runs=runs,
|
||||||
|
signals=signals,
|
||||||
|
latest_report=report,
|
||||||
|
topics=topics,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/logs/recent", response_model=list[str])
|
||||||
|
def recent_logs(settings: Settings = Depends(get_settings)) -> list[str]:
|
||||||
|
# 最近日志用于定位Agent和模型供应商调用问题。
|
||||||
|
if not settings.log_file.exists():
|
||||||
|
return []
|
||||||
|
lines = settings.log_file.read_text(encoding="utf-8", errors="replace").splitlines()
|
||||||
|
return lines[-settings.log_tail_lines :]
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/topics", response_model=TopicRead)
|
||||||
|
def create_topic(payload: TopicCreate, db: Session = Depends(get_db)) -> Topic:
|
||||||
|
# 主题让Agent可以在不改代码的情况下扩展关注范围。
|
||||||
|
topic = Topic(name=payload.name, description=payload.description, enabled=payload.enabled)
|
||||||
|
db.add(topic)
|
||||||
|
db.commit()
|
||||||
|
db.refresh(topic)
|
||||||
|
return topic
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/topics", response_model=list[TopicRead])
|
||||||
|
def list_topics(db: Session = Depends(get_db)) -> list[Topic]:
|
||||||
|
# 主题列表让操作者确认当前情报目标。
|
||||||
|
return list(db.scalars(select(Topic).order_by(Topic.id.asc())).all())
|
||||||
|
|
||||||
|
|
||||||
|
def _visible_signals_query():
|
||||||
|
# 运行中的信号和完成后的信号走同一张表,查询函数保留统一排序入口。
|
||||||
|
return select(Signal).join(AgentRun, Signal.run_id == AgentRun.id)
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Core application settings and wiring."""
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
from functools import lru_cache
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from pydantic import Field
|
||||||
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||||
|
|
||||||
|
|
||||||
|
class Settings(BaseSettings):
|
||||||
|
# 配置统一来自环境变量,保证本地和服务器使用同一套启动路径。
|
||||||
|
app_name: str = "SignalScout"
|
||||||
|
env: str = "dev"
|
||||||
|
database_url: str = Field(
|
||||||
|
default="mysql+pymysql://signalscout:signalscout@127.0.0.1:3306/signalscout?charset=utf8mb4"
|
||||||
|
)
|
||||||
|
llm_base_url: str = "https://api.zayuapi.com/v1"
|
||||||
|
llm_api_key: str = ""
|
||||||
|
llm_model: str = "grok-4.3"
|
||||||
|
llm_timeout_seconds: int = 90
|
||||||
|
llm_max_tokens: int = 4000
|
||||||
|
news_recent_days: int = 3
|
||||||
|
news_max_items: int = 12
|
||||||
|
news_search_max_records: int = 20
|
||||||
|
github_recent_days: int = 30
|
||||||
|
github_min_stars: int = 20
|
||||||
|
github_max_projects: int = 5
|
||||||
|
agent_timezone: str = "Asia/Shanghai"
|
||||||
|
agent_cron_hour: int = 8
|
||||||
|
agent_cron_minute: int = 30
|
||||||
|
report_dir: Path = Path("reports")
|
||||||
|
log_file: Path = Path("logs/app.log")
|
||||||
|
log_tail_lines: int = 200
|
||||||
|
|
||||||
|
model_config = SettingsConfigDict(
|
||||||
|
env_file=".env",
|
||||||
|
env_prefix="SIGNALSCOUT_",
|
||||||
|
extra="ignore",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache
|
||||||
|
def get_settings() -> Settings:
|
||||||
|
# Caching prevents repeated env parsing while tests can still clear it if needed.
|
||||||
|
return Settings()
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
import logging
|
||||||
|
from logging.handlers import RotatingFileHandler
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def configure_logging(log_file: Path) -> None:
|
||||||
|
"""Configure console and file logging once for local service runs."""
|
||||||
|
log_file.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
root_logger = logging.getLogger()
|
||||||
|
root_logger.setLevel(logging.INFO)
|
||||||
|
|
||||||
|
if not _has_handler("signalscout-console"):
|
||||||
|
# Console logs are for the terminal window that runs `make run`.
|
||||||
|
console_handler = logging.StreamHandler()
|
||||||
|
console_handler.name = "signalscout-console"
|
||||||
|
console_handler.setFormatter(_formatter())
|
||||||
|
root_logger.addHandler(console_handler)
|
||||||
|
|
||||||
|
if not _has_handler("signalscout-file"):
|
||||||
|
# File logs persist agent failures after the terminal scrollback is gone.
|
||||||
|
file_handler = RotatingFileHandler(
|
||||||
|
log_file,
|
||||||
|
maxBytes=2_000_000,
|
||||||
|
backupCount=3,
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
file_handler.name = "signalscout-file"
|
||||||
|
file_handler.setFormatter(_formatter())
|
||||||
|
root_logger.addHandler(file_handler)
|
||||||
|
|
||||||
|
|
||||||
|
def _formatter() -> logging.Formatter:
|
||||||
|
# Include module name so source, API, and runner failures are easy to separate.
|
||||||
|
return logging.Formatter("%(asctime)s %(levelname)s [%(name)s] %(message)s")
|
||||||
|
|
||||||
|
|
||||||
|
def _has_handler(name: str) -> bool:
|
||||||
|
# Uvicorn imports the app during startup, so handlers must be idempotent.
|
||||||
|
return any(handler.name == name for handler in logging.getLogger().handlers)
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Database models, sessions, and repositories."""
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, JSON, String, Text, func
|
||||||
|
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
|
||||||
|
|
||||||
|
|
||||||
|
class Base(DeclarativeBase):
|
||||||
|
"""Shared SQLAlchemy declarative base for app models and Alembic metadata."""
|
||||||
|
|
||||||
|
|
||||||
|
class Topic(Base):
|
||||||
|
__tablename__ = "topics"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
name: Mapped[str] = mapped_column(String(120), nullable=False, unique=True)
|
||||||
|
description: Mapped[str] = mapped_column(Text, nullable=False)
|
||||||
|
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||||
|
created_at: Mapped[datetime] = mapped_column(DateTime, server_default=func.now(), nullable=False)
|
||||||
|
updated_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime, server_default=func.now(), onupdate=func.now(), nullable=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class AgentRun(Base):
|
||||||
|
__tablename__ = "agent_runs"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
status: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||||
|
started_at: Mapped[datetime] = mapped_column(DateTime, server_default=func.now(), nullable=False)
|
||||||
|
finished_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
|
||||||
|
error: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
|
||||||
|
request_payload: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||||
|
raw_response: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||||
|
|
||||||
|
signals: Mapped[list["Signal"]] = relationship(back_populates="run")
|
||||||
|
report: Mapped[Optional[Report]] = relationship(back_populates="run", uselist=False)
|
||||||
|
|
||||||
|
|
||||||
|
class Signal(Base):
|
||||||
|
__tablename__ = "signals"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
run_id: Mapped[int] = mapped_column(ForeignKey("agent_runs.id"), nullable=False, index=True)
|
||||||
|
topic: Mapped[str] = mapped_column(String(120), nullable=False, index=True)
|
||||||
|
title: Mapped[str] = mapped_column(String(300), nullable=False)
|
||||||
|
summary: Mapped[str] = mapped_column(Text, nullable=False)
|
||||||
|
source_url: Mapped[str] = mapped_column(Text, nullable=False)
|
||||||
|
source_url_hash: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
|
||||||
|
source_name: Mapped[str] = mapped_column(String(200), nullable=False)
|
||||||
|
published_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
|
||||||
|
importance: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
|
entities: Mapped[list[str]] = mapped_column(JSON, nullable=False, default=list)
|
||||||
|
created_at: Mapped[datetime] = mapped_column(DateTime, server_default=func.now(), nullable=False)
|
||||||
|
|
||||||
|
run: Mapped[AgentRun] = relationship(back_populates="signals")
|
||||||
|
|
||||||
|
|
||||||
|
class Report(Base):
|
||||||
|
__tablename__ = "reports"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
|
run_id: Mapped[int] = mapped_column(ForeignKey("agent_runs.id"), nullable=False)
|
||||||
|
title: Mapped[str] = mapped_column(String(200), nullable=False)
|
||||||
|
content_md: Mapped[str] = mapped_column(Text, nullable=False)
|
||||||
|
file_path: Mapped[Optional[str]] = mapped_column(String(1024), nullable=True)
|
||||||
|
created_at: Mapped[datetime] = mapped_column(DateTime, server_default=func.now(), nullable=False)
|
||||||
|
|
||||||
|
run: Mapped[AgentRun] = relationship(back_populates="report")
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from hashlib import sha256
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.db.models import AgentRun, Report, Signal, Topic
|
||||||
|
from app.schemas.agent import SignalItem
|
||||||
|
|
||||||
|
|
||||||
|
class AgentRepository:
|
||||||
|
"""Database boundary for the agent runner."""
|
||||||
|
|
||||||
|
def __init__(self, db: Session) -> None:
|
||||||
|
self.db = db
|
||||||
|
|
||||||
|
def list_enabled_topics(self) -> list[Topic]:
|
||||||
|
# 启用主题是Agent每次运行时使用的长期目标。
|
||||||
|
return list(self.db.scalars(select(Topic).where(Topic.enabled.is_(True))).all())
|
||||||
|
|
||||||
|
def create_run(self, request_payload: dict) -> AgentRun:
|
||||||
|
# 运行先进入running状态,情报和日报保存完成后再标记完成。
|
||||||
|
run = AgentRun(status="running", request_payload=request_payload)
|
||||||
|
self.db.add(run)
|
||||||
|
self.db.flush()
|
||||||
|
return run
|
||||||
|
|
||||||
|
def mark_run_complete(self, run: AgentRun, raw_response: dict) -> None:
|
||||||
|
run.status = "completed"
|
||||||
|
run.finished_at = datetime.utcnow()
|
||||||
|
run.raw_response = raw_response
|
||||||
|
self.db.flush()
|
||||||
|
|
||||||
|
def mark_run_failed(self, run: AgentRun, error: str) -> None:
|
||||||
|
run.status = "failed"
|
||||||
|
run.finished_at = datetime.utcnow()
|
||||||
|
run.error = error
|
||||||
|
self.db.flush()
|
||||||
|
|
||||||
|
def save_signals(self, run: AgentRun, items: list[SignalItem]) -> list[Signal]:
|
||||||
|
# 相同来源链接只保存一次,避免日报和仪表盘重复出现同一条情报。
|
||||||
|
saved: list[Signal] = []
|
||||||
|
for item in items:
|
||||||
|
signal = self.save_or_update_signal(run=run, item=item)
|
||||||
|
if signal is None:
|
||||||
|
continue
|
||||||
|
saved.append(signal)
|
||||||
|
return saved
|
||||||
|
|
||||||
|
def save_or_update_signal(self, run: AgentRun, item: SignalItem) -> Optional[Signal]:
|
||||||
|
# 同一来源链接只保留一条情报,流式接口和普通接口共用这条去重规则。
|
||||||
|
source_url_hash = self._source_url_hash(item.source_url)
|
||||||
|
existing = self.db.scalar(select(Signal).where(Signal.source_url_hash == source_url_hash))
|
||||||
|
if existing and existing.run_id != run.id:
|
||||||
|
return None
|
||||||
|
if existing:
|
||||||
|
existing.topic = item.topic
|
||||||
|
existing.title = item.title
|
||||||
|
existing.summary = item.summary
|
||||||
|
existing.source_name = item.source_name
|
||||||
|
existing.published_at = item.published_at
|
||||||
|
existing.importance = item.importance
|
||||||
|
existing.entities = item.entities
|
||||||
|
self.db.flush()
|
||||||
|
return existing
|
||||||
|
signal = Signal(
|
||||||
|
run_id=run.id,
|
||||||
|
topic=item.topic,
|
||||||
|
title=item.title,
|
||||||
|
summary=item.summary,
|
||||||
|
source_url=item.source_url,
|
||||||
|
source_url_hash=source_url_hash,
|
||||||
|
source_name=item.source_name,
|
||||||
|
published_at=item.published_at,
|
||||||
|
importance=item.importance,
|
||||||
|
entities=item.entities,
|
||||||
|
)
|
||||||
|
self.db.add(signal)
|
||||||
|
self.db.flush()
|
||||||
|
return signal
|
||||||
|
|
||||||
|
def _source_url_hash(self, source_url: str) -> str:
|
||||||
|
# MySQL不适合直接唯一索引长URL,使用SHA-256作为去重键。
|
||||||
|
return sha256(source_url.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
def save_report(self, run: AgentRun, title: str, content_md: str, file_path: str) -> Report:
|
||||||
|
# 文件路径便于人工打开Markdown产物,MySQL保存可查询正文。
|
||||||
|
report = Report(run_id=run.id, title=title, content_md=content_md, file_path=file_path)
|
||||||
|
self.db.add(report)
|
||||||
|
self.db.flush()
|
||||||
|
return report
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
from collections.abc import Generator
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine
|
||||||
|
from sqlalchemy.orm import Session, sessionmaker
|
||||||
|
|
||||||
|
from app.core.config import get_settings
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
|
||||||
|
# pool_pre_ping避免调度器长时间空闲后拿到失效的MySQL连接。
|
||||||
|
engine = create_engine(settings.database_url, pool_pre_ping=True, future=True)
|
||||||
|
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False, future=True)
|
||||||
|
|
||||||
|
|
||||||
|
def get_db() -> Generator[Session, None, None]:
|
||||||
|
# FastAPI依赖负责在请求结束后关闭数据库会话。
|
||||||
|
db = SessionLocal()
|
||||||
|
try:
|
||||||
|
yield db
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""External data-source clients used by the agent pipeline."""
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from datetime import date, datetime, timedelta
|
||||||
|
from typing import Any, Optional
|
||||||
|
from urllib.error import HTTPError, URLError
|
||||||
|
from urllib.parse import urlencode
|
||||||
|
from urllib.request import Request, urlopen
|
||||||
|
|
||||||
|
from app.core.config import Settings
|
||||||
|
from app.schemas.agent import SignalItem
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class GitHubHotProjectClient:
|
||||||
|
"""Collects recently active, high-signal GitHub AI repositories."""
|
||||||
|
|
||||||
|
endpoint = "https://api.github.com/search/repositories"
|
||||||
|
|
||||||
|
def __init__(self, settings: Settings) -> None:
|
||||||
|
self.settings = settings
|
||||||
|
|
||||||
|
def collect_hot_projects(self, run_date: date) -> list[SignalItem]:
|
||||||
|
# GitHub官方搜索提供开源项目热度候选,最终是否入选交给模型判断。
|
||||||
|
since = run_date - timedelta(days=self.settings.github_recent_days)
|
||||||
|
repositories: dict[str, dict[str, Any]] = {}
|
||||||
|
for query in self._queries(since=since):
|
||||||
|
payload = self._search(query)
|
||||||
|
for repository in payload.get("items", []):
|
||||||
|
html_url = repository.get("html_url")
|
||||||
|
if isinstance(html_url, str):
|
||||||
|
repositories[html_url] = repository
|
||||||
|
|
||||||
|
ranked = sorted(repositories.values(), key=lambda item: self._heat_score(item, run_date), reverse=True)
|
||||||
|
projects = [self._to_signal_item(repository) for repository in ranked[: self.settings.github_max_projects]]
|
||||||
|
logger.info(
|
||||||
|
"github_hot_projects_collected days=%s min_stars=%s projects=%s",
|
||||||
|
self.settings.github_recent_days,
|
||||||
|
self.settings.github_min_stars,
|
||||||
|
len(projects),
|
||||||
|
)
|
||||||
|
return projects
|
||||||
|
|
||||||
|
def _queries(self, since: date) -> list[str]:
|
||||||
|
# 查询集聚焦新近创建的AI项目,避免长期头部仓库挤占每日发现。
|
||||||
|
since_text = since.isoformat()
|
||||||
|
min_stars = self.settings.github_min_stars
|
||||||
|
return [
|
||||||
|
f"topic:llm created:>={since_text} stars:>={min_stars}",
|
||||||
|
f"topic:ai-agent created:>={since_text} stars:>={min_stars}",
|
||||||
|
f"topic:rag created:>={since_text} stars:>={min_stars}",
|
||||||
|
f"llm in:name,description created:>={since_text} stars:>={min_stars}",
|
||||||
|
f"llm agent in:name,description created:>={since_text} stars:>={min_stars}",
|
||||||
|
]
|
||||||
|
|
||||||
|
def _search(self, query: str) -> dict[str, Any]:
|
||||||
|
# GitHub REST搜索直接返回排序后的仓库元数据,便于后续模型研判。
|
||||||
|
url = f"{self.endpoint}?{urlencode({'q': query, 'sort': 'stars', 'order': 'desc', 'per_page': '10'})}"
|
||||||
|
request = Request(
|
||||||
|
url,
|
||||||
|
headers={
|
||||||
|
"Accept": "application/vnd.github+json",
|
||||||
|
"User-Agent": "SignalScout",
|
||||||
|
"X-GitHub-Api-Version": "2022-11-28",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urlopen(request, timeout=20) as response:
|
||||||
|
body = response.read().decode("utf-8")
|
||||||
|
except HTTPError as exc:
|
||||||
|
message = exc.read().decode("utf-8", errors="replace")
|
||||||
|
raise RuntimeError(f"GitHub project search failed: HTTP {exc.code} {message}") from exc
|
||||||
|
except URLError as exc:
|
||||||
|
raise RuntimeError(f"GitHub project search failed: {exc.reason}") from exc
|
||||||
|
return json.loads(body)
|
||||||
|
|
||||||
|
def _to_signal_item(self, repository: dict[str, Any]) -> SignalItem:
|
||||||
|
# 仓库元数据先标准化为信号候选,后续由模型统一筛选和改写摘要。
|
||||||
|
full_name = str(repository.get("full_name") or repository.get("name") or "Unknown repository")
|
||||||
|
description = str(repository.get("description") or "暂无项目描述")
|
||||||
|
stars = int(repository.get("stargazers_count") or 0)
|
||||||
|
forks = int(repository.get("forks_count") or 0)
|
||||||
|
language = repository.get("language") or "未标注语言"
|
||||||
|
created_at = self._parse_datetime(repository.get("created_at"))
|
||||||
|
pushed_at = self._parse_datetime(repository.get("pushed_at"))
|
||||||
|
freshness = created_at.date().isoformat() if created_at else "最近"
|
||||||
|
summary = (
|
||||||
|
f"{description} 该项目创建于 {freshness},当前约 {stars} stars、{forks} forks,"
|
||||||
|
f"主要语言为 {language}。"
|
||||||
|
)
|
||||||
|
return SignalItem(
|
||||||
|
topic="GitHub 热门项目",
|
||||||
|
title=full_name,
|
||||||
|
summary=summary,
|
||||||
|
source_url=str(repository["html_url"]),
|
||||||
|
source_name="GitHub",
|
||||||
|
published_at=created_at or pushed_at,
|
||||||
|
importance=3,
|
||||||
|
entities=[full_name, "GitHub", str(language)],
|
||||||
|
)
|
||||||
|
|
||||||
|
def _heat_score(self, repository: dict[str, Any], run_date: date) -> int:
|
||||||
|
# Stars表示关注度,forks表示采用迹象,新项目获得额外新鲜度权重。
|
||||||
|
stars = int(repository.get("stargazers_count") or 0)
|
||||||
|
forks = int(repository.get("forks_count") or 0)
|
||||||
|
created_at = self._parse_datetime(repository.get("created_at"))
|
||||||
|
recency_bonus = 100 if created_at and created_at.date() >= run_date - timedelta(days=7) else 0
|
||||||
|
return stars * 3 + forks + recency_bonus
|
||||||
|
|
||||||
|
def _parse_datetime(self, value: Any) -> Optional[datetime]:
|
||||||
|
# GitHub时间戳使用UTC ISO格式和Z后缀。
|
||||||
|
if not isinstance(value, str):
|
||||||
|
return None
|
||||||
|
return datetime.fromisoformat(value.replace("Z", "+00:00")).replace(tzinfo=None)
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from datetime import date, datetime, timedelta
|
||||||
|
from typing import Any, Optional
|
||||||
|
from urllib.error import HTTPError, URLError
|
||||||
|
from urllib.parse import urlencode
|
||||||
|
from urllib.request import Request, urlopen
|
||||||
|
|
||||||
|
from app.core.config import Settings
|
||||||
|
from app.schemas.agent import SignalItem
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class NewsSearchClient:
|
||||||
|
"""通过公开新闻搜索接口采集动态AI新闻候选。"""
|
||||||
|
|
||||||
|
endpoint = "https://api.gdeltproject.org/api/v2/doc/doc"
|
||||||
|
|
||||||
|
def __init__(self, settings: Settings) -> None:
|
||||||
|
self.settings = settings
|
||||||
|
|
||||||
|
def collect_ai_news(self, run_date: date) -> list[SignalItem]:
|
||||||
|
# GDELT按日期窗口返回全球新闻候选,模型后续只从这些证据里挑选日报条目。
|
||||||
|
since = run_date - timedelta(days=self.settings.news_recent_days)
|
||||||
|
articles: dict[str, dict[str, Any]] = {}
|
||||||
|
payload = self._search(query=self._query(), since=since, until=run_date)
|
||||||
|
for article in payload.get("articles", []):
|
||||||
|
url = article.get("url")
|
||||||
|
if isinstance(url, str) and url.startswith("http"):
|
||||||
|
articles[url] = article
|
||||||
|
|
||||||
|
candidates = [self._article_to_signal(article) for article in articles.values()]
|
||||||
|
candidates = [candidate for candidate in candidates if candidate is not None]
|
||||||
|
candidates.sort(key=lambda item: item.published_at or datetime.min, reverse=True)
|
||||||
|
logger.info("news_candidates_collected days=%s items=%s", self.settings.news_recent_days, len(candidates))
|
||||||
|
return candidates[: self.settings.news_search_max_records]
|
||||||
|
|
||||||
|
def _query(self) -> str:
|
||||||
|
# GDELT限制请求频率,单次组合查询覆盖模型、Agent、开源和融资四类情报。
|
||||||
|
return (
|
||||||
|
'("AI agent" OR "agent framework" OR "LLM agent" OR '
|
||||||
|
'"large language model" OR LLM OR "AI model" OR '
|
||||||
|
'"open source AI" OR "open-source AI" OR "AI startup funding")'
|
||||||
|
)
|
||||||
|
|
||||||
|
def _search(self, query: str, since: date, until: date) -> dict[str, Any]:
|
||||||
|
# ArtList模式返回标题、URL、域名和发现时间,正好满足候选证据采集。
|
||||||
|
params = {
|
||||||
|
"query": query,
|
||||||
|
"mode": "ArtList",
|
||||||
|
"format": "json",
|
||||||
|
"maxrecords": str(self.settings.news_search_max_records),
|
||||||
|
"sort": "HybridRel",
|
||||||
|
"startdatetime": since.strftime("%Y%m%d000000"),
|
||||||
|
"enddatetime": until.strftime("%Y%m%d235959"),
|
||||||
|
}
|
||||||
|
url = f"{self.endpoint}?{urlencode(params)}"
|
||||||
|
request = Request(url, headers={"User-Agent": "SignalScout"})
|
||||||
|
try:
|
||||||
|
with urlopen(request, timeout=30) as response:
|
||||||
|
body = response.read().decode("utf-8")
|
||||||
|
except HTTPError as exc:
|
||||||
|
message = exc.read().decode("utf-8", errors="replace")
|
||||||
|
raise RuntimeError(f"News search failed: HTTP {exc.code} {message}") from exc
|
||||||
|
except URLError as exc:
|
||||||
|
raise RuntimeError(f"News search failed: {exc.reason}") from exc
|
||||||
|
return json.loads(body)
|
||||||
|
|
||||||
|
def _article_to_signal(self, article: dict[str, Any]) -> Optional[SignalItem]:
|
||||||
|
# 新闻候选先保留标题、链接和来源,重要性与摘要由模型最终确定。
|
||||||
|
title = article.get("title")
|
||||||
|
url = article.get("url")
|
||||||
|
if not isinstance(title, str) or not isinstance(url, str):
|
||||||
|
return None
|
||||||
|
domain = str(article.get("domain") or "News")
|
||||||
|
source_country = str(article.get("sourcecountry") or "")
|
||||||
|
published_at = self._parse_gdelt_date(article.get("seendate"))
|
||||||
|
return SignalItem(
|
||||||
|
topic="AI 新闻候选",
|
||||||
|
title=title[:300],
|
||||||
|
summary=f"来自 {domain} 的新闻候选,GDELT发现时间为 {article.get('seendate') or '未知'}。",
|
||||||
|
source_url=url,
|
||||||
|
source_name=domain,
|
||||||
|
published_at=published_at,
|
||||||
|
importance=3,
|
||||||
|
entities=[value for value in [domain, source_country] if value],
|
||||||
|
)
|
||||||
|
|
||||||
|
def _parse_gdelt_date(self, value: Any) -> Optional[datetime]:
|
||||||
|
# GDELT使用类似20260708T120000Z的时间格式。
|
||||||
|
if not isinstance(value, str):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return datetime.strptime(value, "%Y%m%dT%H%M%SZ")
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""模型调用与结构化情报生成模块。"""
|
||||||
@@ -0,0 +1,198 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from datetime import date, timedelta
|
||||||
|
from typing import Any
|
||||||
|
from urllib.error import HTTPError, URLError
|
||||||
|
from urllib.request import Request, urlopen
|
||||||
|
|
||||||
|
from app.core.config import Settings
|
||||||
|
from app.db.models import Topic
|
||||||
|
from app.schemas.agent import AgentResult, SignalItem
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class GrokIntelligenceClient:
|
||||||
|
"""通过 OpenAI 兼容接口调用 Grok,生成每日 AI 情报结构化结果。"""
|
||||||
|
|
||||||
|
def __init__(self, settings: Settings) -> None:
|
||||||
|
self.settings = settings
|
||||||
|
|
||||||
|
def collect_daily_intelligence(
|
||||||
|
self,
|
||||||
|
topics: list[Topic],
|
||||||
|
news_candidates: list[SignalItem],
|
||||||
|
github_projects: list[SignalItem],
|
||||||
|
run_date: date,
|
||||||
|
) -> AgentResult:
|
||||||
|
# 先用确定性 GitHub 候选缩小开源项目范围,再由模型统一判断新闻价值和日报结构。
|
||||||
|
if not self.settings.llm_api_key:
|
||||||
|
raise RuntimeError("SIGNALSCOUT_LLM_API_KEY is required")
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": self._system_prompt()},
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": self._user_prompt(
|
||||||
|
topics=topics,
|
||||||
|
news_candidates=news_candidates,
|
||||||
|
github_projects=github_projects,
|
||||||
|
run_date=run_date,
|
||||||
|
),
|
||||||
|
},
|
||||||
|
]
|
||||||
|
logger.info(
|
||||||
|
"grok_intelligence_started model=%s run_date=%s github_candidates=%s",
|
||||||
|
self.settings.llm_model,
|
||||||
|
run_date.isoformat(),
|
||||||
|
len(github_projects),
|
||||||
|
)
|
||||||
|
payload = self._chat_json(messages)
|
||||||
|
result = AgentResult.model_validate(payload)
|
||||||
|
logger.info(
|
||||||
|
"grok_intelligence_completed model=%s signals=%s",
|
||||||
|
self.settings.llm_model,
|
||||||
|
len(result.signals),
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def _chat_json(self, messages: list[dict[str, str]]) -> dict[str, Any]:
|
||||||
|
# JSON mode把结构化契约交给模型侧执行,解析失败时直接暴露供应商原文便于排障。
|
||||||
|
url = f"{self.settings.llm_base_url.rstrip('/')}/chat/completions"
|
||||||
|
body = json.dumps(
|
||||||
|
{
|
||||||
|
"model": self.settings.llm_model,
|
||||||
|
"messages": messages,
|
||||||
|
"temperature": 0.2,
|
||||||
|
"max_tokens": self.settings.llm_max_tokens,
|
||||||
|
"response_format": {"type": "json_object"},
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
).encode("utf-8")
|
||||||
|
request = Request(
|
||||||
|
url,
|
||||||
|
data=body,
|
||||||
|
headers={
|
||||||
|
"Authorization": f"Bearer {self.settings.llm_api_key}",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"Accept": "application/json",
|
||||||
|
},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urlopen(request, timeout=self.settings.llm_timeout_seconds) as response:
|
||||||
|
response_body = response.read().decode("utf-8")
|
||||||
|
except HTTPError as exc:
|
||||||
|
message = exc.read().decode("utf-8", errors="replace")
|
||||||
|
raise RuntimeError(f"Grok request failed: HTTP {exc.code} {message}") from exc
|
||||||
|
except URLError as exc:
|
||||||
|
raise RuntimeError(f"Grok request failed: {exc.reason}") from exc
|
||||||
|
|
||||||
|
completion = json.loads(response_body)
|
||||||
|
content = completion["choices"][0]["message"]["content"]
|
||||||
|
if not isinstance(content, str):
|
||||||
|
raise RuntimeError("Grok response content is not text")
|
||||||
|
try:
|
||||||
|
payload = json.loads(content)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise RuntimeError(f"Grok returned invalid JSON: {content[:1200]}") from exc
|
||||||
|
payload.setdefault("raw_response", {})
|
||||||
|
payload["raw_response"].update(
|
||||||
|
{
|
||||||
|
"provider": self.settings.llm_base_url,
|
||||||
|
"model": self.settings.llm_model,
|
||||||
|
"usage": completion.get("usage", {}),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return payload
|
||||||
|
|
||||||
|
def _system_prompt(self) -> str:
|
||||||
|
# 这段提示词定义唯一产出格式,数据库与报告都依赖同一个模型结果。
|
||||||
|
return (
|
||||||
|
"你是 SignalScout,一个专门追踪 AI 新闻、模型发布、Agent 工具、开源项目和融资动态的情报 Agent。"
|
||||||
|
"你必须优先给出最近发生、可点击验证、对工程和产品判断有价值的事件。"
|
||||||
|
"所有结论必须带 source_url,不能编造链接。"
|
||||||
|
"只输出一个 JSON object,不输出 Markdown 代码块或额外解释。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _user_prompt(
|
||||||
|
self,
|
||||||
|
topics: list[Topic],
|
||||||
|
news_candidates: list[SignalItem],
|
||||||
|
github_projects: list[SignalItem],
|
||||||
|
run_date: date,
|
||||||
|
) -> str:
|
||||||
|
# 日期窗口让模型把“最新”落到明确范围,GitHub候选则避免热门项目只靠语言模型记忆。
|
||||||
|
since = run_date - timedelta(days=self.settings.news_recent_days)
|
||||||
|
topic_payload = [
|
||||||
|
{"name": topic.name, "description": topic.description}
|
||||||
|
for topic in topics
|
||||||
|
if topic.enabled
|
||||||
|
]
|
||||||
|
github_payload = [
|
||||||
|
{
|
||||||
|
"title": item.title,
|
||||||
|
"summary": item.summary,
|
||||||
|
"source_url": item.source_url,
|
||||||
|
"published_at": item.published_at.isoformat() if item.published_at else None,
|
||||||
|
"entities": item.entities,
|
||||||
|
}
|
||||||
|
for item in github_projects
|
||||||
|
]
|
||||||
|
news_payload = [
|
||||||
|
{
|
||||||
|
"title": item.title,
|
||||||
|
"summary": item.summary,
|
||||||
|
"source_url": item.source_url,
|
||||||
|
"source_name": item.source_name,
|
||||||
|
"published_at": item.published_at.isoformat() if item.published_at else None,
|
||||||
|
"entities": item.entities,
|
||||||
|
}
|
||||||
|
for item in news_candidates
|
||||||
|
]
|
||||||
|
return json.dumps(
|
||||||
|
{
|
||||||
|
"task": "生成每日 AI 情报日报,并返回结构化信号。",
|
||||||
|
"run_date": run_date.isoformat(),
|
||||||
|
"date_window": {
|
||||||
|
"from": since.isoformat(),
|
||||||
|
"to": run_date.isoformat(),
|
||||||
|
},
|
||||||
|
"limits": {
|
||||||
|
"max_news_signals": self.settings.news_max_items,
|
||||||
|
"max_github_signals": self.settings.github_max_projects,
|
||||||
|
},
|
||||||
|
"topics": topic_payload,
|
||||||
|
"news_candidates": news_payload,
|
||||||
|
"github_candidates": github_payload,
|
||||||
|
"output_schema": {
|
||||||
|
"title": "字符串,日报标题",
|
||||||
|
"signals": [
|
||||||
|
{
|
||||||
|
"topic": "AI 新闻 / GitHub 热门项目 / 模型发布 / Agent 工具 / 融资动态等",
|
||||||
|
"title": "字符串",
|
||||||
|
"summary": "中文摘要,说明为什么重要",
|
||||||
|
"source_url": "可点击来源链接",
|
||||||
|
"source_name": "来源名称",
|
||||||
|
"published_at": "ISO 时间;不知道具体时间可用日期T00:00:00",
|
||||||
|
"importance": "1到5的整数",
|
||||||
|
"entities": ["公司、项目、模型、人名等实体"],
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"report_markdown": "中文 Markdown 日报,包含今日重点、AI 新闻、GitHub 热门项目和观察",
|
||||||
|
"raw_response": {
|
||||||
|
"notes": "简短说明检索和筛选依据",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"rules": [
|
||||||
|
"新闻必须来自日期窗口内或接近日内发生的事件。",
|
||||||
|
"AI新闻只能从 news_candidates 中选择,不要新增候选外新闻。",
|
||||||
|
"GitHub 热门项目只能从 github_candidates 中选择,不要新增候选外仓库。",
|
||||||
|
"每条 signal 必须有真实 source_url。",
|
||||||
|
"report_markdown 只能基于 signals 写,不要加入 signals 外的新事实。",
|
||||||
|
"输出必须是可被 json.loads 解析的 JSON object。",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
)
|
||||||
+53
@@ -0,0 +1,53 @@
|
|||||||
|
import sys
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
import uvicorn
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
|
||||||
|
# Direct IDE runs should not leave Python bytecode caches in the project tree.
|
||||||
|
sys.dont_write_bytecode = True
|
||||||
|
|
||||||
|
from app.agent.scheduler import build_scheduler # noqa: E402
|
||||||
|
from app.api.routes import router # noqa: E402
|
||||||
|
from app.core.config import get_settings # noqa: E402
|
||||||
|
from app.core.log_setup import configure_logging # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||||
|
# The scheduler lives with the web process for the demo deployment profile.
|
||||||
|
settings = get_settings()
|
||||||
|
scheduler = build_scheduler(settings)
|
||||||
|
scheduler.start()
|
||||||
|
app.state.scheduler = scheduler
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
scheduler.shutdown(wait=False)
|
||||||
|
|
||||||
|
|
||||||
|
def create_app() -> FastAPI:
|
||||||
|
# App factory keeps tests and production startup on the same initialization path.
|
||||||
|
settings = get_settings()
|
||||||
|
configure_logging(settings.log_file)
|
||||||
|
app = FastAPI(title=settings.app_name, lifespan=lifespan)
|
||||||
|
# The separate Vite frontend runs on 5173 during local development.
|
||||||
|
app.add_middleware(
|
||||||
|
CORSMiddleware,
|
||||||
|
allow_origins=["http://127.0.0.1:5173", "http://localhost:5173"],
|
||||||
|
allow_credentials=False,
|
||||||
|
allow_methods=["*"],
|
||||||
|
allow_headers=["*"],
|
||||||
|
)
|
||||||
|
app.include_router(router)
|
||||||
|
return app
|
||||||
|
|
||||||
|
|
||||||
|
app = create_app()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
# Direct execution is for IDE/local development; deployment can still call uvicorn explicitly.
|
||||||
|
uvicorn.run("app.main:app", host="127.0.0.1", port=8010, reload=False)
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Pydantic contracts used across the app."""
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
|
||||||
|
class SearchTopic(BaseModel):
|
||||||
|
# 主题是模型每次运行时使用的长期情报目标。
|
||||||
|
name: str
|
||||||
|
description: str
|
||||||
|
|
||||||
|
|
||||||
|
class SignalItem(BaseModel):
|
||||||
|
# 模型产出的结构化情报会通过这个契约进入数据库和日报。
|
||||||
|
topic: str
|
||||||
|
title: str
|
||||||
|
summary: str
|
||||||
|
source_url: str
|
||||||
|
source_name: str
|
||||||
|
published_at: Optional[datetime] = None
|
||||||
|
importance: int = Field(ge=1, le=5)
|
||||||
|
entities: list[str] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class AgentResult(BaseModel):
|
||||||
|
# Agent结果同时承载机器可读信号和人可读日报。
|
||||||
|
title: str
|
||||||
|
signals: list[SignalItem]
|
||||||
|
report_markdown: str
|
||||||
|
raw_response: dict = Field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class RunResponse(BaseModel):
|
||||||
|
run_id: int
|
||||||
|
status: str
|
||||||
|
report_path: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
class RunRead(BaseModel):
|
||||||
|
id: int
|
||||||
|
status: str
|
||||||
|
started_at: datetime
|
||||||
|
finished_at: Optional[datetime]
|
||||||
|
error: Optional[str]
|
||||||
|
|
||||||
|
model_config = {"from_attributes": True}
|
||||||
|
|
||||||
|
|
||||||
|
class SignalRead(BaseModel):
|
||||||
|
id: int
|
||||||
|
run_id: int
|
||||||
|
topic: str
|
||||||
|
title: str
|
||||||
|
summary: str
|
||||||
|
source_url: str
|
||||||
|
source_name: str
|
||||||
|
published_at: Optional[datetime]
|
||||||
|
importance: int
|
||||||
|
entities: list[str]
|
||||||
|
created_at: datetime
|
||||||
|
|
||||||
|
model_config = {"from_attributes": True}
|
||||||
|
|
||||||
|
|
||||||
|
class ReportRead(BaseModel):
|
||||||
|
id: int
|
||||||
|
run_id: int
|
||||||
|
title: str
|
||||||
|
content_md: str
|
||||||
|
file_path: Optional[str]
|
||||||
|
created_at: datetime
|
||||||
|
|
||||||
|
model_config = {"from_attributes": True}
|
||||||
|
|
||||||
|
|
||||||
|
class DashboardRead(BaseModel):
|
||||||
|
runs: list[RunRead]
|
||||||
|
signals: list[SignalRead]
|
||||||
|
latest_report: Optional[ReportRead]
|
||||||
|
topics: list["TopicRead"]
|
||||||
|
|
||||||
|
|
||||||
|
class TopicCreate(BaseModel):
|
||||||
|
name: str = Field(min_length=2, max_length=120)
|
||||||
|
description: str = Field(min_length=5)
|
||||||
|
enabled: bool = True
|
||||||
|
|
||||||
|
|
||||||
|
class TopicRead(BaseModel):
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
description: str
|
||||||
|
enabled: bool
|
||||||
|
|
||||||
|
model_config = {"from_attributes": True}
|
||||||
|
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
repo_url="${REPO_URL:-http://gitadmin:12345678911@gitea:3000/gitadmin/signalscout-backend.git}"
|
||||||
|
source_dir="/opt/signalscout-source"
|
||||||
|
remote_dir="/opt/signalscout"
|
||||||
|
release_dir="/opt/signalscout.release"
|
||||||
|
public_ready_url="${PUBLIC_READY_URL:-}"
|
||||||
|
python_bin="${PYTHON_BIN:-python3.9}"
|
||||||
|
deploy_lock="/tmp/signalscout-backend-deploy.lock"
|
||||||
|
|
||||||
|
# 同一时间只允许一个部署任务替换源码、虚拟环境和运行目录,避免并发构建互相删除文件。
|
||||||
|
exec 9>"${deploy_lock}"
|
||||||
|
flock 9
|
||||||
|
|
||||||
|
# 服务器直接从 Gitea main 分支取代码,Jenkins和手动部署共享同一条发布路径。
|
||||||
|
if [ ! -d "${source_dir}/.git" ]; then
|
||||||
|
rm -rf "${source_dir}"
|
||||||
|
git clone --branch main "${repo_url}" "${source_dir}"
|
||||||
|
else
|
||||||
|
git -C "${source_dir}" remote set-url origin "${repo_url}"
|
||||||
|
git -C "${source_dir}" fetch origin main
|
||||||
|
git -C "${source_dir}" reset --hard origin/main
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! command -v "${python_bin}" >/dev/null 2>&1; then
|
||||||
|
echo "Python runtime is required but ${python_bin} was not found" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 源码目录先安装开发依赖并跑测试,测试通过后才生成发布目录。
|
||||||
|
rm -rf "${source_dir}/.venv"
|
||||||
|
"${python_bin}" -m venv "${source_dir}/.venv"
|
||||||
|
"${source_dir}/.venv/bin/python" -m ensurepip --upgrade
|
||||||
|
"${source_dir}/.venv/bin/python" -m pip install -e "${source_dir}[dev]"
|
||||||
|
"${source_dir}/.venv/bin/python" -B -m ruff check "${source_dir}"
|
||||||
|
"${source_dir}/.venv/bin/python" -B -m pytest "${source_dir}"
|
||||||
|
|
||||||
|
# 发布目录重新生成,避免把源码仓库、测试缓存或旧虚拟环境带入 systemd 运行目录。
|
||||||
|
rm -rf "${release_dir}"
|
||||||
|
mkdir -p "${release_dir}"
|
||||||
|
tar \
|
||||||
|
--exclude=.git \
|
||||||
|
--exclude=.venv \
|
||||||
|
--exclude=.pytest_cache \
|
||||||
|
--exclude=.ruff_cache \
|
||||||
|
--exclude=__pycache__ \
|
||||||
|
--exclude=*.pyc \
|
||||||
|
--exclude=.env \
|
||||||
|
--exclude=reports \
|
||||||
|
--exclude=logs \
|
||||||
|
-C "${source_dir}" \
|
||||||
|
-cf - . | tar -C "${release_dir}" -xf -
|
||||||
|
|
||||||
|
# 服务配置以服务器当前 .env 为准;缺失时直接失败,避免 systemd 启动后才暴露为502。
|
||||||
|
if [ ! -f "${remote_dir}/.env" ]; then
|
||||||
|
echo "signalscout backend .env is missing at ${remote_dir}/.env" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
cp "${remote_dir}/.env" "${release_dir}/.env"
|
||||||
|
|
||||||
|
"${python_bin}" -m venv "${release_dir}/.venv"
|
||||||
|
"${release_dir}/.venv/bin/python" -m ensurepip --upgrade
|
||||||
|
"${release_dir}/.venv/bin/python" -m pip install -e "${release_dir}"
|
||||||
|
(
|
||||||
|
cd "${release_dir}"
|
||||||
|
"${release_dir}/.venv/bin/python" -B -m alembic upgrade head
|
||||||
|
"${release_dir}/.venv/bin/python" -B scripts/seed_topics.py
|
||||||
|
)
|
||||||
|
|
||||||
|
rm -rf "${remote_dir}"
|
||||||
|
mv "${release_dir}" "${remote_dir}"
|
||||||
|
mkdir -p "${remote_dir}/logs" "${remote_dir}/reports"
|
||||||
|
|
||||||
|
cp "${remote_dir}/deployment/systemd/signalscout.service" /etc/systemd/system/signalscout.service
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl restart signalscout
|
||||||
|
if ! systemctl is-active --quiet signalscout; then
|
||||||
|
systemctl status signalscout --no-pager -l >&2 || true
|
||||||
|
journalctl -u signalscout -n 120 --no-pager >&2 || true
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# systemd启动成功后继续等待HTTP健康检查,确保API真实开始监听。
|
||||||
|
for attempt in $(seq 1 20); do
|
||||||
|
if curl -fsS "http://127.0.0.1:8020/health" >/dev/null; then
|
||||||
|
if [ -n "${public_ready_url}" ]; then
|
||||||
|
curl -fsS "${public_ready_url}" >/dev/null
|
||||||
|
fi
|
||||||
|
echo "signalscout backend deploy ok"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "signalscout backend health check failed" >&2
|
||||||
|
systemctl status signalscout --no-pager -l >&2 || true
|
||||||
|
journalctl -u signalscout -n 120 --no-pager >&2 || true
|
||||||
|
exit 1
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=SignalScout AI Intelligence API
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
WorkingDirectory=/opt/signalscout
|
||||||
|
EnvironmentFile=/opt/signalscout/.env
|
||||||
|
ExecStart=/opt/signalscout/.venv/bin/python -B -m uvicorn app.main:app --host 127.0.0.1 --port 8020
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# SignalScout 部署
|
||||||
|
|
||||||
|
SignalScout 使用和 `agent-desk` 一致的 Gitea + Jenkins 发布链路。
|
||||||
|
|
||||||
|
## 服务器路径
|
||||||
|
|
||||||
|
- Gitea 仓库:`gitadmin/signalscout-backend`
|
||||||
|
- Jenkins Job:`signalscout-backend`
|
||||||
|
- 源码目录:`/opt/signalscout-source`
|
||||||
|
- 发布目录:`/opt/signalscout`
|
||||||
|
- systemd 服务:`signalscout`
|
||||||
|
- 本机监听:`127.0.0.1:8020`
|
||||||
|
- 线上域名:`https://signalscout.jaycode.online`
|
||||||
|
|
||||||
|
## 流水线
|
||||||
|
|
||||||
|
Jenkins 从 Gitea 仓库读取根目录 `Jenkinsfile`,然后通过已有凭据 SSH 到服务器执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
deployment/jenkins/deploy_backend.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
脚本负责:
|
||||||
|
|
||||||
|
- 从 Gitea 拉取 `main` 分支
|
||||||
|
- 创建测试虚拟环境
|
||||||
|
- 执行 `ruff` 和 `pytest`
|
||||||
|
- 生成发布目录
|
||||||
|
- 执行 Alembic 迁移和主题种子初始化
|
||||||
|
- 重启 `signalscout` systemd 服务
|
||||||
|
- 检查本机和公网健康接口
|
||||||
|
|
||||||
|
## Nginx 与 Cloudflare Tunnel
|
||||||
|
|
||||||
|
域名通过 Cloudflare Tunnel 进入服务器,Nginx 监听本机端口:
|
||||||
|
|
||||||
|
```nginx
|
||||||
|
server {
|
||||||
|
listen 127.0.0.1:8360;
|
||||||
|
server_name signalscout.jaycode.online;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://127.0.0.1:8020;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Cloudflare Tunnel ingress 指向:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- hostname: signalscout.jaycode.online
|
||||||
|
service: http://127.0.0.1:8360
|
||||||
|
```
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
from logging.config import fileConfig
|
||||||
|
|
||||||
|
from alembic import context
|
||||||
|
from sqlalchemy import engine_from_config, pool
|
||||||
|
|
||||||
|
from app.core.config import get_settings
|
||||||
|
from app.db.models import Base
|
||||||
|
|
||||||
|
config = context.config
|
||||||
|
|
||||||
|
if config.config_file_name is not None:
|
||||||
|
fileConfig(config.config_file_name)
|
||||||
|
|
||||||
|
target_metadata = Base.metadata
|
||||||
|
|
||||||
|
|
||||||
|
def get_url() -> str:
|
||||||
|
# Alembic reads the same settings object as the app so schema changes target one database.
|
||||||
|
return get_settings().database_url
|
||||||
|
|
||||||
|
|
||||||
|
def run_migrations_offline() -> None:
|
||||||
|
# Offline mode renders SQL without opening a database connection.
|
||||||
|
context.configure(
|
||||||
|
url=get_url(),
|
||||||
|
target_metadata=target_metadata,
|
||||||
|
literal_binds=True,
|
||||||
|
dialect_opts={"paramstyle": "named"},
|
||||||
|
)
|
||||||
|
|
||||||
|
with context.begin_transaction():
|
||||||
|
context.run_migrations()
|
||||||
|
|
||||||
|
|
||||||
|
def run_migrations_online() -> None:
|
||||||
|
# Online mode is used by local dev and docker compose to apply migrations directly.
|
||||||
|
configuration = config.get_section(config.config_ini_section) or {}
|
||||||
|
configuration["sqlalchemy.url"] = get_url()
|
||||||
|
connectable = engine_from_config(
|
||||||
|
configuration,
|
||||||
|
prefix="sqlalchemy.",
|
||||||
|
poolclass=pool.NullPool,
|
||||||
|
)
|
||||||
|
|
||||||
|
with connectable.connect() as connection:
|
||||||
|
context.configure(connection=connection, target_metadata=target_metadata)
|
||||||
|
|
||||||
|
with context.begin_transaction():
|
||||||
|
context.run_migrations()
|
||||||
|
|
||||||
|
|
||||||
|
if context.is_offline_mode():
|
||||||
|
run_migrations_offline()
|
||||||
|
else:
|
||||||
|
run_migrations_online()
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
"""${message}
|
||||||
|
|
||||||
|
Revision ID: ${up_revision}
|
||||||
|
Revises: ${down_revision | comma,n}
|
||||||
|
Create Date: ${create_date}
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
${imports if imports else ""}
|
||||||
|
|
||||||
|
revision: str = ${repr(up_revision)}
|
||||||
|
down_revision: Union[str, None] = ${repr(down_revision)}
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
|
||||||
|
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
${upgrades if upgrades else "pass"}
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
${downgrades if downgrades else "pass"}
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
"""initial schema
|
||||||
|
|
||||||
|
Revision ID: 202605071200
|
||||||
|
Revises:
|
||||||
|
Create Date: 2026-05-07 12:00:00
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
revision: str = "202605071200"
|
||||||
|
down_revision: Union[str, None] = None
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
# Topics define durable intelligence goals instead of one-off prompts.
|
||||||
|
op.create_table(
|
||||||
|
"topics",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("name", sa.String(length=120), nullable=False, unique=True),
|
||||||
|
sa.Column("description", sa.Text(), nullable=False),
|
||||||
|
sa.Column("enabled", sa.Boolean(), nullable=False, server_default=sa.true()),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
sa.Column("updated_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Runs are the audit trail for every scheduled or manual agent execution.
|
||||||
|
op.create_table(
|
||||||
|
"agent_runs",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("status", sa.String(length=32), nullable=False),
|
||||||
|
sa.Column("started_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
sa.Column("finished_at", sa.DateTime(), nullable=True),
|
||||||
|
sa.Column("error", sa.Text(), nullable=True),
|
||||||
|
sa.Column("request_payload", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("raw_response", sa.JSON(), nullable=True),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Signals store normalized facts that the reporter and future memory layer can query.
|
||||||
|
op.create_table(
|
||||||
|
"signals",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("run_id", sa.Integer(), sa.ForeignKey("agent_runs.id"), nullable=False),
|
||||||
|
sa.Column("topic", sa.String(length=120), nullable=False),
|
||||||
|
sa.Column("title", sa.String(length=300), nullable=False),
|
||||||
|
sa.Column("summary", sa.Text(), nullable=False),
|
||||||
|
sa.Column("source_url", sa.Text(), nullable=False),
|
||||||
|
sa.Column("source_url_hash", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("source_name", sa.String(length=200), nullable=False),
|
||||||
|
sa.Column("published_at", sa.DateTime(), nullable=True),
|
||||||
|
sa.Column("importance", sa.Integer(), nullable=False),
|
||||||
|
sa.Column("entities", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
sa.UniqueConstraint("source_url_hash", name="uq_signals_source_url_hash"),
|
||||||
|
)
|
||||||
|
op.create_index("ix_signals_run_id", "signals", ["run_id"])
|
||||||
|
op.create_index("ix_signals_topic", "signals", ["topic"])
|
||||||
|
|
||||||
|
# Reports are persisted both in MySQL and as markdown files for human reading.
|
||||||
|
op.create_table(
|
||||||
|
"reports",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("run_id", sa.Integer(), sa.ForeignKey("agent_runs.id"), nullable=False),
|
||||||
|
sa.Column("title", sa.String(length=200), nullable=False),
|
||||||
|
sa.Column("content_md", sa.Text(), nullable=False),
|
||||||
|
sa.Column("file_path", sa.String(length=1024), nullable=True),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_table("reports")
|
||||||
|
op.drop_index("ix_signals_topic", table_name="signals")
|
||||||
|
op.drop_index("ix_signals_run_id", table_name="signals")
|
||||||
|
op.drop_table("signals")
|
||||||
|
op.drop_table("agent_runs")
|
||||||
|
op.drop_table("topics")
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
"""source subscriptions and raw items
|
||||||
|
|
||||||
|
Revision ID: 202605080900
|
||||||
|
Revises: 202605071200
|
||||||
|
Create Date: 2026-05-08 09:00:00
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
revision: str = "202605080900"
|
||||||
|
down_revision: Union[str, None] = "202605071200"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
# Subscriptions are stable source inputs that run before model-side search.
|
||||||
|
op.create_table(
|
||||||
|
"source_subscriptions",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("name", sa.String(length=160), nullable=False, unique=True),
|
||||||
|
sa.Column("source_type", sa.String(length=32), nullable=False),
|
||||||
|
sa.Column("url", sa.Text(), nullable=False),
|
||||||
|
sa.Column("url_hash", sa.String(length=64), nullable=False, unique=True),
|
||||||
|
sa.Column("category", sa.String(length=120), nullable=False),
|
||||||
|
sa.Column("enabled", sa.Boolean(), nullable=False, server_default=sa.true()),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
sa.Column("updated_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Raw items preserve source evidence before the final signal layer deduplicates and summarizes it.
|
||||||
|
op.create_table(
|
||||||
|
"raw_items",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("run_id", sa.Integer(), sa.ForeignKey("agent_runs.id"), nullable=False),
|
||||||
|
sa.Column(
|
||||||
|
"subscription_id",
|
||||||
|
sa.Integer(),
|
||||||
|
sa.ForeignKey("source_subscriptions.id"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.Column("title", sa.String(length=300), nullable=False),
|
||||||
|
sa.Column("summary", sa.Text(), nullable=False),
|
||||||
|
sa.Column("source_url", sa.Text(), nullable=False),
|
||||||
|
sa.Column("source_url_hash", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("source_name", sa.String(length=200), nullable=False),
|
||||||
|
sa.Column("category", sa.String(length=120), nullable=False),
|
||||||
|
sa.Column("published_at", sa.DateTime(), nullable=True),
|
||||||
|
sa.Column("raw_payload", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
sa.UniqueConstraint("source_url_hash", name="uq_raw_items_source_url_hash"),
|
||||||
|
)
|
||||||
|
op.create_index("ix_raw_items_run_id", "raw_items", ["run_id"])
|
||||||
|
op.create_index("ix_raw_items_subscription_id", "raw_items", ["subscription_id"])
|
||||||
|
op.create_index("ix_raw_items_category", "raw_items", ["category"])
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_index("ix_raw_items_category", table_name="raw_items")
|
||||||
|
op.drop_index("ix_raw_items_subscription_id", table_name="raw_items")
|
||||||
|
op.drop_index("ix_raw_items_run_id", table_name="raw_items")
|
||||||
|
op.drop_table("raw_items")
|
||||||
|
op.drop_table("source_subscriptions")
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
"""drop subscription tables
|
||||||
|
|
||||||
|
Revision ID: 202607080900
|
||||||
|
Revises: 202605080900
|
||||||
|
Create Date: 2026-07-08 09:00:00
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
revision: str = "202607080900"
|
||||||
|
down_revision: Union[str, None] = "202605080900"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
# Agent主路径已经改为模型结构化情报,历史订阅源表从运行模型中移除。
|
||||||
|
op.drop_index("ix_raw_items_category", table_name="raw_items")
|
||||||
|
op.drop_index("ix_raw_items_subscription_id", table_name="raw_items")
|
||||||
|
op.drop_index("ix_raw_items_run_id", table_name="raw_items")
|
||||||
|
op.drop_table("raw_items")
|
||||||
|
op.drop_table("source_subscriptions")
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
# 回滚时恢复旧表结构,便于数据库迁移链保持可逆。
|
||||||
|
op.create_table(
|
||||||
|
"source_subscriptions",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("name", sa.String(length=160), nullable=False, unique=True),
|
||||||
|
sa.Column("source_type", sa.String(length=32), nullable=False),
|
||||||
|
sa.Column("url", sa.Text(), nullable=False),
|
||||||
|
sa.Column("url_hash", sa.String(length=64), nullable=False, unique=True),
|
||||||
|
sa.Column("category", sa.String(length=120), nullable=False),
|
||||||
|
sa.Column("enabled", sa.Boolean(), nullable=False, server_default=sa.true()),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
sa.Column("updated_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
)
|
||||||
|
op.create_table(
|
||||||
|
"raw_items",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True),
|
||||||
|
sa.Column("run_id", sa.Integer(), sa.ForeignKey("agent_runs.id"), nullable=False),
|
||||||
|
sa.Column("subscription_id", sa.Integer(), sa.ForeignKey("source_subscriptions.id"), nullable=False),
|
||||||
|
sa.Column("title", sa.String(length=300), nullable=False),
|
||||||
|
sa.Column("summary", sa.Text(), nullable=False),
|
||||||
|
sa.Column("source_url", sa.Text(), nullable=False),
|
||||||
|
sa.Column("source_url_hash", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("source_name", sa.String(length=200), nullable=False),
|
||||||
|
sa.Column("category", sa.String(length=120), nullable=False),
|
||||||
|
sa.Column("published_at", sa.DateTime(), nullable=True),
|
||||||
|
sa.Column("raw_payload", sa.JSON(), nullable=False),
|
||||||
|
sa.Column("created_at", sa.DateTime(), nullable=False, server_default=sa.func.now()),
|
||||||
|
sa.UniqueConstraint("source_url_hash", name="uq_raw_items_source_url_hash"),
|
||||||
|
)
|
||||||
|
op.create_index("ix_raw_items_run_id", "raw_items", ["run_id"])
|
||||||
|
op.create_index("ix_raw_items_subscription_id", "raw_items", ["subscription_id"])
|
||||||
|
op.create_index("ix_raw_items_category", "raw_items", ["category"])
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
[project]
|
||||||
|
name = "signalscout"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "A scheduled AI intelligence agent powered by Grok and GitHub search."
|
||||||
|
requires-python = ">=3.9,<3.13"
|
||||||
|
dependencies = [
|
||||||
|
"alembic==1.13.3",
|
||||||
|
"apscheduler==3.10.4",
|
||||||
|
"fastapi==0.115.6",
|
||||||
|
"pydantic==2.10.4",
|
||||||
|
"pydantic-settings==2.7.1",
|
||||||
|
"pymysql==1.1.1",
|
||||||
|
"python-dotenv==1.0.1",
|
||||||
|
"sqlalchemy==2.0.36",
|
||||||
|
"uvicorn[standard]==0.34.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
dev = [
|
||||||
|
"pytest==8.3.4",
|
||||||
|
"ruff==0.8.6",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
line-length = 100
|
||||||
|
target-version = "py39"
|
||||||
|
|
||||||
|
[tool.setuptools.packages.find]
|
||||||
|
include = ["app*"]
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
testpaths = ["tests"]
|
||||||
|
pythonpath = ["."]
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
from app.db.models import Topic
|
||||||
|
from app.db.session import SessionLocal
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULT_TOPICS = [
|
||||||
|
{
|
||||||
|
"name": "Large models and agents",
|
||||||
|
"description": "New model releases, agent frameworks, tooling, benchmarks, and product launches.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Open-source AI projects",
|
||||||
|
"description": "Important GitHub projects, open model releases, framework updates, and adoption signals.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "AI products and funding",
|
||||||
|
"description": "AI product launches, startup funding, acquisitions, and strategic market movement.",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
# 种子数据只创建模型研判时使用的长期主题,采集来源由Agent运行时动态完成。
|
||||||
|
db = SessionLocal()
|
||||||
|
try:
|
||||||
|
for item in DEFAULT_TOPICS:
|
||||||
|
exists = db.query(Topic).filter(Topic.name == item["name"]).first()
|
||||||
|
if exists:
|
||||||
|
continue
|
||||||
|
db.add(Topic(**item, enabled=True))
|
||||||
|
db.commit()
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from app.main import create_app
|
||||||
|
|
||||||
|
|
||||||
|
def test_health_endpoint_returns_app_identity() -> None:
|
||||||
|
# 健康检查只暴露服务身份,不触发Agent运行或数据库写入。
|
||||||
|
with TestClient(create_app()) as client:
|
||||||
|
response = client.get("/health")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json()["status"] == "ok"
|
||||||
|
assert response.json()["app"] == "SignalScout"
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
from app.core.config import Settings
|
||||||
|
from app.integrations.github_client import GitHubHotProjectClient
|
||||||
|
|
||||||
|
|
||||||
|
def test_github_client_maps_repository_to_signal() -> None:
|
||||||
|
# GitHub仓库候选先标准化为信号,再交给模型做最终筛选。
|
||||||
|
client = GitHubHotProjectClient(Settings())
|
||||||
|
signal = client._to_signal_item(
|
||||||
|
{
|
||||||
|
"full_name": "example/agent-kit",
|
||||||
|
"description": "Agent framework for production workflows.",
|
||||||
|
"html_url": "https://github.com/example/agent-kit",
|
||||||
|
"stargazers_count": 2400,
|
||||||
|
"forks_count": 180,
|
||||||
|
"language": "Python",
|
||||||
|
"created_at": "2026-05-01T09:00:00Z",
|
||||||
|
"pushed_at": "2026-05-07T10:00:00Z",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert signal.topic == "GitHub 热门项目"
|
||||||
|
assert signal.source_name == "GitHub"
|
||||||
|
assert signal.source_url == "https://github.com/example/agent-kit"
|
||||||
|
assert "2400 stars" in signal.summary
|
||||||
|
assert "2026-05-01" in signal.summary
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
from datetime import date, datetime
|
||||||
|
|
||||||
|
from app.core.config import Settings
|
||||||
|
from app.db.models import Topic
|
||||||
|
from app.llm.grok_client import GrokIntelligenceClient
|
||||||
|
from app.schemas.agent import SignalItem
|
||||||
|
|
||||||
|
|
||||||
|
class StubGrokClient(GrokIntelligenceClient):
|
||||||
|
def _chat_json(self, messages):
|
||||||
|
# 测试只验证提示词输入和结果契约,不访问真实模型服务。
|
||||||
|
assert "github_candidates" in messages[1]["content"]
|
||||||
|
return {
|
||||||
|
"title": "AI 情报日报 2026-07-08",
|
||||||
|
"signals": [
|
||||||
|
{
|
||||||
|
"topic": "GitHub 热门项目",
|
||||||
|
"title": "example/agent-kit",
|
||||||
|
"summary": "该项目提供生产级 Agent 工作流能力。",
|
||||||
|
"source_url": "https://github.com/example/agent-kit",
|
||||||
|
"source_name": "GitHub",
|
||||||
|
"published_at": "2026-07-08T00:00:00",
|
||||||
|
"importance": 4,
|
||||||
|
"entities": ["example/agent-kit", "Agent"],
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"report_markdown": "# AI 情报日报 2026-07-08",
|
||||||
|
"raw_response": {"notes": "基于模型检索和 GitHub 候选筛选。"},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_grok_client_returns_agent_result() -> None:
|
||||||
|
# Grok客户端是日报与结构化情报的唯一生成入口。
|
||||||
|
client = StubGrokClient(Settings(llm_api_key="test-key"))
|
||||||
|
result = client.collect_daily_intelligence(
|
||||||
|
topics=[
|
||||||
|
Topic(
|
||||||
|
id=1,
|
||||||
|
name="Open-source AI projects",
|
||||||
|
description="Agent frameworks and open-source AI tools.",
|
||||||
|
enabled=True,
|
||||||
|
)
|
||||||
|
],
|
||||||
|
news_candidates=[
|
||||||
|
SignalItem(
|
||||||
|
topic="AI 新闻候选",
|
||||||
|
title="New model release",
|
||||||
|
summary="News candidate from example.com.",
|
||||||
|
source_url="https://example.com/model-release",
|
||||||
|
source_name="example.com",
|
||||||
|
published_at=datetime(2026, 7, 8),
|
||||||
|
importance=3,
|
||||||
|
entities=["example.com"],
|
||||||
|
)
|
||||||
|
],
|
||||||
|
github_projects=[
|
||||||
|
SignalItem(
|
||||||
|
topic="GitHub 热门项目",
|
||||||
|
title="example/agent-kit",
|
||||||
|
summary="Agent framework.",
|
||||||
|
source_url="https://github.com/example/agent-kit",
|
||||||
|
source_name="GitHub",
|
||||||
|
published_at=datetime(2026, 7, 8),
|
||||||
|
importance=4,
|
||||||
|
entities=["Agent"],
|
||||||
|
)
|
||||||
|
],
|
||||||
|
run_date=date(2026, 7, 8),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.title == "AI 情报日报 2026-07-08"
|
||||||
|
assert result.signals[0].source_name == "GitHub"
|
||||||
|
assert result.report_markdown.startswith("# AI 情报日报")
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
from app.core.config import Settings
|
||||||
|
from app.integrations.news_search_client import NewsSearchClient
|
||||||
|
|
||||||
|
|
||||||
|
def test_news_search_client_maps_article_to_candidate() -> None:
|
||||||
|
# 新闻搜索结果会先转为候选信号,再交给模型筛选成最终情报。
|
||||||
|
client = NewsSearchClient(Settings())
|
||||||
|
signal = client._article_to_signal(
|
||||||
|
{
|
||||||
|
"title": "AI agent platform launches",
|
||||||
|
"url": "https://example.com/ai-agent-platform",
|
||||||
|
"domain": "example.com",
|
||||||
|
"sourcecountry": "US",
|
||||||
|
"seendate": "20260708T120000Z",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert signal is not None
|
||||||
|
assert signal.topic == "AI 新闻候选"
|
||||||
|
assert signal.source_name == "example.com"
|
||||||
|
assert signal.published_at is not None
|
||||||
|
assert signal.published_at.year == 2026
|
||||||
Reference in New Issue
Block a user