aicraft

package module
v0.0.0-...-4425940 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 21, 2024 License: CC0-1.0 Imports: 12 Imported by: 0

README

AICraft: Workflow Automation with AI

AICraft is a Go package that automates workflows using predefined tools, such as OpenAI's GPT-4 for content generation, DALL·E for image generation, and text-embedding-ada-002 for generating embeddings from PDF content. This package allows you to create agents, assign tasks, and execute complex workflows involving AI-driven tools.

Features

  • PDF to Embeddings Conversion: Convert PDF content into embeddings using OpenAI's text-embedding-ada-002.
  • Content Generation: Generate or optimize content using OpenAI's GPT-4.
  • Image Generation: Create images or diagrams using OpenAI's DALL·E.
  • Workflow Automation: Chain tasks together into workflows with dependency management.

Installation

To install the package, simply run:

go get github.com/DevMaan707/aicraft

Tools

  • PDFToEmbeddingsTool: Converts PDF content into embeddings using OpenAI's text-embedding-ada-002.

  • OpenAIContentGeneratorTool: Generates or optimizes content using OpenAI's GPT-4.

  • ImageGeneratorTool: Generates images or diagrams using OpenAI's DALL·E.

  • TextToPDFTool: Converts generated text and images into a PDF.

Agents

Agents are responsible for executing tasks. Each agent can depend on other agents, ensuring tasks are executed in the correct order.

Documentation

Key Components

  1. Manager:

    • Responsible for creating agents, tasks, and tools.
    • Manages the execution of workflows and ensures tasks are executed in the correct order.

  2. Agent:

    • An entity that executes one or more tasks.
    • Can have dependencies on other agents to ensure proper execution order.

  3. Task:

    • Represents a unit of work that uses a specific tool.
    • Takes input parameters and produces output after execution.

  4. Tool:

    • A predefined function or API call used to perform specific tasks, such as content generation or image creation.
Key Functions

  1. NewManager(): Creates a new Manager instance.

  2. CreateAgent(id, name string, dependsOn []string): Creates a new Agent with a unique ID, name, and optional dependencies.

  3. CreateTask(id, name, toolID string, inputs map[string]interface{}): Creates a new Task with a unique ID, name, and associated tool.

  4. AssignTaskToAgent(agentID, taskID string): Assigns a task to an agent.

  5. ExecuteWorkflow(): Executes all agents and tasks within the manager, respecting dependencies.

Predefined Tools

  1. PDFToEmbeddingsTool:

    • Converts PDF content into embeddings using OpenAI's text-embedding-ada-002 model.
    • Inputs: pdf_content (string), api_key (string).

  2. OpenAIContentGeneratorTool:

    • Generates or optimizes content using OpenAI's GPT-4 model.
    • Inputs: query (string), api_key (string).

  3. ImageGeneratorTool:

    • Generates images or diagrams using OpenAI's DALL·E model.
    • Inputs: description (string), api_key (string).

  4. TextToPDFTool:

    • Converts text into a PDF document.
    • Inputs: text (string).
Example Workflow

An example workflow can be set up to convert a PDF into embeddings, optimize a query, generate related images, and compile everything into a final PDF document.

Refer to the README.md for a step-by-step example of how to create and execute such a workflow.

Advanced Features
  • Dependency Management: Agents can depend on other agents, allowing for complex workflows where tasks are executed in a specific order.
  • Error Handling: The ExecuteWorkflow function ensures that errors are properly handled and reported during execution.
Extending AICraft

Users can extend the aicraft package by defining their own tools and tasks. This allows for greater flexibility and customization of workflows.

Conclusion

The aicraft package provides a powerful and flexible way to automate complex workflows involving AI-driven tasks. By leveraging predefined tools and the ability to define dependencies between agents, users can create sophisticated processes that handle everything from text generation to PDF creation.

License

This project is licensed under the MIT License. See the LICENSE file for more details.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	TextToPDFTool = &Tool{
		ID:   "text_to_pdf",
		Name: "Text to PDF",
		Execute: func(inputs map[string]interface{}) (interface{}, <-chan interface{}, error) {
			text, ok := inputs["text"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'text' is required and must be a string")
			}
			fmt.Println("Converting text to PDF...")
			return fmt.Sprintf("PDF Content from: %s", text), nil, nil
		},
	}

	OpenAIContentGeneratorTool = &Tool{
		ID:   "openai_content_generator",
		Name: "OpenAI Content Generator",
		Execute: func(inputs map[string]interface{}) (interface{}, <-chan interface{}, error) {
			query, ok := inputs["query"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'query' is required and must be a string")
			}
			chunkSize, ok := inputs["chunkSize"].(int)
			if !ok {
				return nil, nil, fmt.Errorf("input 'chunkSize' is required and must be an int")
			}
			chunkOverlap, ok := inputs["chunkOverlap"].(int)
			if !ok {
				return nil, nil, fmt.Errorf("input 'chunkOverlap' is required and must be an int")
			}

			context, ok := inputs["context"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'context' is required and must be a string")
			}
			fmt.Println(context)
			apiKey, ok := inputs["api_key"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'api_key' is required and must be a string")
			}

			verbose, _ := inputs["verbose"].(bool)

			model := "gpt-3.5-turbo"
			if m, ok := inputs["model"].(string); ok && m != "" {
				model = m
			}

			contextChunks := SplitTextIntoChunks(context, chunkSize, chunkOverlap)

			prompt := fmt.Sprintf("Context: %s\n\nQuery: %s", contextChunks[0], query)
			if EstimateTokens(prompt) > maxTokens {
				prompt = truncateTextToTokenLimit(prompt, maxTokens-500)
			}

			if verbose {
				log.Printf("Generated prompt: %s", prompt)
			}

			data := map[string]interface{}{
				"model":  model,
				"stream": true,
				"messages": []map[string]string{
					{"role": "user", "content": prompt},
				},
			}

			jsonData, err := json.Marshal(data)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to marshal request data: %v", err)
			}

			req, err := http.NewRequest("POST", "https://api.openai.com/v1/chat/completions", bytes.NewBuffer(jsonData))
			if err != nil {
				return nil, nil, fmt.Errorf("failed to create request: %v", err)
			}

			req.Header.Set("Content-Type", "application/json")
			req.Header.Set("Authorization", "Bearer "+apiKey)

			client := &http.Client{}
			resp, err := client.Do(req)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to execute request: %v", err)
			}

			contentChannel := make(chan interface{})

			go func() {
				defer resp.Body.Close()
				defer close(contentChannel)

				reader := bufio.NewReader(resp.Body)
				for {
					line, err := reader.ReadString('\n')
					if err != nil {
						if err == io.EOF {
							break
						}
						log.Printf("failed to read stream: %v", err)
						break
					}

					if strings.TrimSpace(line) == "data: [DONE]" {
						break
					}

					if strings.HasPrefix(line, "data: ") {
						line = line[len("data: "):]
					}

					var streamResponse struct {
						Choices []struct {
							Delta struct {
								Content string `json:"content"`
							} `json:"delta"`
						} `json:"choices"`
					}

					if err := json.Unmarshal([]byte(line), &streamResponse); err != nil {
						continue
					}

					if len(streamResponse.Choices) > 0 {
						content := streamResponse.Choices[0].Delta.Content

						contentChannel <- content
					}
				}
			}()

			return nil, contentChannel, nil
		},
	}

	ImageGeneratorTool = &Tool{
		ID:   "image_generator",
		Name: "Image Generator",
		Execute: func(inputs map[string]interface{}) (interface{}, <-chan interface{}, error) {
			description, ok := inputs["description"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'description' is required and must be a string")
			}

			apiKey, ok := inputs["api_key"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'api_key' is required and must be a string")
			}

			verbose, _ := inputs["verbose"].(bool)

			data := map[string]interface{}{
				"prompt": description,
				"n":      1,
				"size":   "1024x1024",
			}

			if verbose {
				log.Printf("Generating image with description: %s", description)
			}

			jsonData, err := json.Marshal(data)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to marshal request data: %v", err)
			}

			req, err := http.NewRequest("POST", "https://api.openai.com/v1/images/generations", bytes.NewBuffer(jsonData))
			if err != nil {
				return nil, nil, fmt.Errorf("failed to create request: %v", err)
			}

			req.Header.Set("Content-Type", "application/json")
			req.Header.Set("Authorization", "Bearer "+apiKey)

			client := &http.Client{}
			resp, err := client.Do(req)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to execute request: %v", err)
			}
			defer resp.Body.Close()

			var response struct {
				Data []struct {
					URL string `json:"url"`
				} `json:"data"`
			}
			err = json.NewDecoder(resp.Body).Decode(&response)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to decode response: %v", err)
			}

			if len(response.Data) == 0 {
				return nil, nil, fmt.Errorf("no images returned from OpenAI API")
			}

			if verbose {
				log.Printf("Generated image URL: %s", response.Data[0].URL)
			}

			return response.Data[0].URL, nil, nil
		},
	}

	QueryToEmbeddingTool = &Tool{
		ID:   "query_to_embedding",
		Name: "Query to Embedding",
		Execute: func(inputs map[string]interface{}) (interface{}, <-chan interface{}, error) {
			query, ok := inputs["query"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'query' is required and must be a string")
			}

			apiKey, ok := inputs["api_key"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'api_key' is required and must be a string")
			}

			verbose, _ := inputs["verbose"].(bool)

			model := "text-embedding-ada-002"
			if m, ok := inputs["model"].(string); ok && m != "" {
				model = m
			}

			data := map[string]interface{}{
				"model": model,
				"input": query,
			}

			jsonData, err := json.Marshal(data)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to marshal request data: %v", err)
			}

			req, err := http.NewRequest("POST", "https://api.openai.com/v1/embeddings", bytes.NewBuffer(jsonData))
			if err != nil {
				return nil, nil, fmt.Errorf("failed to create request: %v", err)
			}

			req.Header.Set("Content-Type", "application/json")
			req.Header.Set("Authorization", "Bearer "+apiKey)

			if verbose {
				log.Printf("Sending query to OpenAI Embedding API: %s", query)
			}

			client := &http.Client{}
			resp, err := client.Do(req)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to execute request: %v", err)
			}
			defer resp.Body.Close()

			var response struct {
				Data []struct {
					Embedding []float64 `json:"embedding"`
				} `json:"data"`
			}

			err = json.NewDecoder(resp.Body).Decode(&response)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to decode response: %v", err)
			}

			if len(response.Data) == 0 {
				return nil, nil, fmt.Errorf("no embeddings returned from OpenAI API")
			}

			if verbose {
				log.Printf("Query embedding generated successfully, embedding length: %d", len(response.Data[0].Embedding))
			}

			return response.Data[0].Embedding, nil, nil
		},
	}

	PDFToEmbeddingsTool = &Tool{
		ID:   "pdf_to_embeddings",
		Name: "PDF to Embeddings",
		Execute: func(inputs map[string]interface{}) (interface{}, <-chan interface{}, error) {
			pdfContent, _ := inputs["pdf_content"].(string)
			log.Println("PDF CONTENT LENGTH => " + fmt.Sprintf("%d", len(pdfContent)))
			chunkSize, _ := inputs["chunkSize"].(int)
			chunkOverlap, ok := inputs["chunkOverlap"].(int)
			if !ok {
				return nil, nil, fmt.Errorf("input 'chunkOverlap' is required and must be an int")
			}

			apiKey, ok := inputs["api_key"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'api_key' is required and must be a string")
			}

			verbose, _ := inputs["verbose"].(bool)

			chunks := SplitTextIntoChunks(pdfContent, chunkSize, chunkOverlap)
			var allEmbeddings [][]float64

			for i, chunk := range chunks {
				if verbose {
					log.Printf("Processing chunk %d: %s", i+1, chunk)
				}

				if len(chunk) == 0 {
					if verbose {
						log.Printf("Chunk %d is empty, skipping.", i+1)
					}
					continue
				}

				data := map[string]interface{}{
					"model": "text-embedding-ada-002",
					"input": chunk,
				}

				jsonData, err := json.Marshal(data)
				if err != nil {
					return nil, nil, fmt.Errorf("failed to marshal request data: %v", err)
				}

				req, err := http.NewRequest("POST", "https://api.openai.com/v1/embeddings", bytes.NewBuffer(jsonData))
				if err != nil {
					return nil, nil, fmt.Errorf("failed to create request: %v", err)
				}

				req.Header.Set("Content-Type", "application/json")
				req.Header.Set("Authorization", "Bearer "+apiKey)

				client := &http.Client{}
				resp, err := client.Do(req)
				if err != nil {
					return nil, nil, fmt.Errorf("failed to execute request: %v", err)
				}
				defer resp.Body.Close()

				var response struct {
					Data []struct {
						Embedding []float64 `json:"embedding"`
					} `json:"data"`
				}
				err = json.NewDecoder(resp.Body).Decode(&response)
				if err != nil {
					return nil, nil, fmt.Errorf("failed to decode response: %v", err)
				}

				if len(response.Data) == 0 {
					if verbose {
						log.Printf("Failed to produce embeddings for the provided text chunk: %s", chunk)
					}
					return nil, nil, fmt.Errorf("no embeddings returned from OpenAI API")
				}

				allEmbeddings = append(allEmbeddings, response.Data[0].Embedding)
				if verbose {
					log.Printf("Chunk %d processed successfully, embedding length: %d", i+1, len(response.Data[0].Embedding))
				}
			}

			if len(allEmbeddings) == 0 {
				return nil, nil, fmt.Errorf("no valid embeddings were produced")
			}

			if verbose {
				log.Printf("Total embeddings generated: %d", len(allEmbeddings))
			}

			return allEmbeddings, nil, nil
		},
	}

	PDFExtractorTool = &Tool{
		ID:   "pdf_extractor",
		Name: "PDF Extractor",
		Execute: func(inputs map[string]interface{}) (interface{}, <-chan interface{}, error) {
			pdfURL, ok := inputs["pdf_url"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'pdf_url' is required and must be a string")
			}

			verbose, _ := inputs["verbose"].(bool)

			pdfFilePath, err := DownloadPDF(pdfURL)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to download PDF: %v", err)
			}
			defer os.Remove(pdfFilePath)

			text, err := ExtractTextFromPDF(pdfFilePath)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to extract text from PDF: %v", err)
			}

			if verbose {
				log.Println("Extracted text from PDF:", text)
			}

			return text, nil, nil
		},
	}

	ImageNeedCheckerTool = &Tool{
		ID:   "image_need_checker",
		Name: "Image Need Checker",
		Execute: func(inputs map[string]interface{}) (interface{}, <-chan interface{}, error) {
			content, ok := inputs["content"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'content' is required and must be a string")
			}

			apiKey, ok := inputs["api_key"].(string)
			if !ok {
				return nil, nil, fmt.Errorf("input 'api_key' is required and must be a string")
			}
			model := "text-embedding-ada-002"
			if m, ok := inputs["model"].(string); ok && m != "" {
				model = m
			}
			data := map[string]interface{}{
				"model": model,
				"messages": []map[string]string{
					{"role": "system", "content": "You are an assistant that identifies the need for diagrams or flowcharts in text."},
					{"role": "user", "content": fmt.Sprintf("Given the following content, identify if any diagrams or flowcharts are needed and provide descriptions: %s. THE OUTPUT TO BE STRICTLY A SLICE OF STRINGS OR ARRAY OF STRINGS", content)},
				},
			}

			jsonData, err := json.Marshal(data)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to marshal request data: %v", err)
			}

			req, err := http.NewRequest("POST", "https://api.openai.com/v1/chat/completions", bytes.NewBuffer(jsonData))
			if err != nil {
				return nil, nil, fmt.Errorf("failed to create request: %v", err)
			}

			req.Header.Set("Content-Type", "application/json")
			req.Header.Set("Authorization", "Bearer "+apiKey)

			client := &http.Client{}
			resp, err := client.Do(req)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to execute request: %v", err)
			}
			defer resp.Body.Close()

			var response OpenAIResponse
			err = json.NewDecoder(resp.Body).Decode(&response)
			if err != nil {
				return nil, nil, fmt.Errorf("failed to decode response: %v", err)
			}

			if len(response.Choices) == 0 {
				return nil, nil, fmt.Errorf("no choices returned from OpenAI API")
			}

			return response.Choices[0].Message.Content, nil, nil
		},
	}
)

Functions

func CosineSimilarity 

func CosineSimilarity(vec1, vec2 []float64) float64

func DownloadPDF 

func DownloadPDF(pdfURL string) (string, error)

func EstimateTokens 

func EstimateTokens(text string) int

func ExtractDescriptions 

func ExtractDescriptions(content string) []string

func ExtractRelevantText 

func ExtractRelevantText(extractedText string, index int, chunkSize int) string

func ExtractRelevantText1 

func ExtractRelevantText1(extractedText string, index int, chunkSize int) string

func ExtractTextFromPDF 

func ExtractTextFromPDF(path string) (string, error)

func FindMostSimilarChunk 

func FindMostSimilarChunk(queryEmbedding []float64, docEmbeddings [][]float64) int

func FlattenAndConvertToFloat32 

func FlattenAndConvertToFloat32(embeddings [][]float64) ([]float32, []int)

func ReconstructToFloat64 

func ReconstructToFloat64(flattened []float32, lengths []int) [][]float64

func SplitTextIntoChunks 

func SplitTextIntoChunks(text string, chunkSize int, chunkOverlap int) []string

Types

type Agent 

type Agent struct {
	ID        string
	Name      string
	Tasks     []*Task
	DependsOn []string
	Output    map[string]interface{}
	Stream    <-chan interface{}
}

func NewAgent 

func NewAgent(id, name string, dependsOn []string) *Agent

func (*Agent) AddTask 

func (a *Agent) AddTask(task *Task)

func (*Agent) ExecuteTasks 

func (a *Agent) ExecuteTasks() error

type AgentConfig 

type AgentConfig struct {
	ID        string
	Name      string
	DependsOn []string
	Tasks     []string
}

type Manager 

type Manager struct {
	Agents map[string]*Agent
	Tasks  map[string]*Task
	Tools  map[string]*Tool
	// contains filtered or unexported fields
}

func NewManager 

func NewManager() *Manager

func (*Manager) AssignTaskToAgent 

func (m *Manager) AssignTaskToAgent(agentID, taskID string)

func (*Manager) CreateAgent 

func (m *Manager) CreateAgent(id, name string, dependsOn []string) *Agent

func (*Manager) CreateTask 

func (m *Manager) CreateTask(id, name string, toolID string, inputs map[string]interface{}) *Task

func (*Manager) ExecuteAllWorkflows 

func (m *Manager) ExecuteAllWorkflows() error

func (*Manager) ExecuteWorkflow 

func (m *Manager) ExecuteWorkflow() error

func (*Manager) InitializeWorkflow 

func (m *Manager) InitializeWorkflow(config WorkflowConfig) error

type OpenAIResponse 

type OpenAIResponse struct {
	Choices []struct {
		Message struct {
			Role    string `json:"role"`
			Content string `json:"content"`
		} `json:"message"`
	} `json:"choices"`
}

type Task 

type Task struct {
	ID     string
	Name   string
	Tool   *Tool
	Inputs map[string]interface{}
	Result interface{}
	Stream <-chan interface{}
}

func NewTask 

func NewTask(id, name string, tool *Tool, inputs map[string]interface{}) *Task

func (*Task) Execute 

func (t *Task) Execute() error

type TaskConfig 

type TaskConfig struct {
	ID     string
	Name   string
	ToolID string
	Inputs map[string]interface{}
}

type Tool 

type Tool struct {
	ID      string
	Name    string
	Execute func(inputs map[string]interface{}) (interface{}, <-chan interface{}, error)
}

type WorkflowConfig 

type WorkflowConfig struct {
	Tasks  []TaskConfig
	Agents []AgentConfig
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
test command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.