diff --git a/ADOPTERS.md b/ADOPTERS.md index d4241fab4..8b8173dd6 100644 --- a/ADOPTERS.md +++ b/ADOPTERS.md @@ -1,3 +1,39 @@ -See https://www.bfe-networks.net/en_us/#users . +Please [reply the issue](https://github.com/bfenetworks/bfe/issues/748) if you want to be listed here. -Please [create an issue](https://github.com/bfenetworks/bfe/issues/new?&title=New%20BFE%20User) if you want to be listed there. +Some of our users include: + +* [Baidu](https://www.baidu.com) + +* [CCTV](https://www.cctv.com) + +* [China Life](https://www.chinalife.com.cn) + +* [China Merchants Bank](https://www.cmbchina.com) + +* [Duxiaoman Financial](https://www.duxiaoman.com) + +* [Haier](https://www.haier.com) + +* [Postal Savings Bank of China](https://www.psbc.com) + +* [Resolink](https://www.crresolink.com.cn) + +* [Safesoft](http://www.safesoftcorp.com) + +* [Shenzhen Stock Exchange](http://www.szse.cn) + +* [Sichuan Airlines](https://www.sichuanair.com) + +* [SPD Bank](https://www.spdb.com.cn) + +* [State Grid](http://www.sgcc.com.cn) + +* [Sunbox](http://www.sunboxsoft.com) + +* [USTC](https://www.ustc.edu.cn) + +* [Xixuetong](http://www.xixuetong.com) + +* [Yillion Bank](https://www.yillionbank.com) + +* [360](https://www.so.com) diff --git a/CHANGELOG.md b/CHANGELOG.md index 70ceeb3ef..b3d4e5772 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,16 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [v1.3.0] - 2021-09-16 + +### Added +- Support basic route rules that are similar to k8s ingress rules +- Documents optimization + +### Changed +- Ignore GREASE values for JA3 fingerprint + + ## [v1.2.0] - 2021-06-21 ### Added @@ -252,6 +262,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Flexible plugin framework to extend functionality. Based on the framework, developer can add new features rapidly - Detailed built-in metrics available for service status monitor +[v1.3.0]: https://github.com/bfenetworks/bfe/compare/v1.2.0...v1.3.0 [v1.2.0]: https://github.com/bfenetworks/bfe/compare/v1.1.0...v1.2.0 [v1.1.0]: https://github.com/bfenetworks/bfe/compare/v1.0.0...v1.1.0 [v1.0.0]: https://github.com/bfenetworks/bfe/compare/v0.12.0...v1.0.0 diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 213190d5b..f34dd8f6d 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -70,9 +70,12 @@ | | 0xflotus | | | calify | | | Gii16 | +| | lancoLiu | +| | mingjliu9 | | | MoonShining | | | odidev | | | pirDOL | +| | qloog | | | tianlan2011 | | | u5surf | | | xiaocongwjb | diff --git a/README-CN.md b/README-CN.md new file mode 100644 index 000000000..4978613ac --- /dev/null +++ b/README-CN.md @@ -0,0 +1,72 @@ +# BFE + +[English](README.md) | 中文 + +[![GitHub](https://img.shields.io/github/license/bfenetworks/bfe)](https://github.com/bfenetworks/bfe/blob/develop/LICENSE) +[![Travis](https://img.shields.io/travis/com/bfenetworks/bfe)](https://travis-ci.com/bfenetworks/bfe) +[![Go Report Card](https://goreportcard.com/badge/github.com/bfenetworks/bfe)](https://goreportcard.com/report/github.com/bfenetworks/bfe) +[![GoDoc](https://godoc.org/github.com/bfenetworks/bfe?status.svg)](https://godoc.org/github.com/bfenetworks/bfe/bfe_module) +[![Snap Status](https://build.snapcraft.io/badge/bfenetworks/bfe.svg)](https://build.snapcraft.io/user/bfenetworks/bfe) +[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/3209/badge)](https://bestpractices.coreinfrastructure.org/projects/3209) +[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fbfenetworks%2Fbfe.svg?type=shield)](https://app.fossa.com/reports/1f05f9f0-ac3d-486e-8ba9-ad95dabd4768) +[![CLA assistant](https://cla-assistant.io/readme/badge/bfenetworks/bfe)](https://cla-assistant.io/bfenetworks/bfe) +[![Slack Widget](https://img.shields.io/badge/join-us%20on%20slack-gray.svg?longCache=true&logo=slack&colorB=green)](https://slack.cncf.io) + +BFE是百度开源的现代化七层负载均衡系统 + +## 特性及优点 +- 丰富协议支持：支持HTTP、HTTPS、SPDY、HTTP/2、WebSocket、TLS、gRPC、FastCGI等 +- 基于请求内容的路由：支持高级条件表达式定制转发规则，转发规则易于理解及维护 +- 高级负载均衡：支持全局/分布式负载均衡，实现就近访问、跨可用区容灾及过载保护等 +- 灵活的模块框架：支持高效率定制开发第三方扩展模块 +- 一流的可见性：提供丰富详尽的监控指标，提供各类日志供问题诊断、数据分析及可视化 +[了解更多详情](https://www.bfe-networks.net/zh_cn/introduction/overview/) + +## 开始使用 +- [编译及运行](docs/zh_cn/installation/install_from_source.md) + +## 运行测试 +- 请参考[编译及运行](docs/zh_cn/installation/install_from_source.md) + +## 文档 +- [英文版](https://www.bfe-networks.net/en_us/ABOUT/) +- [中文版](https://www.bfe-networks.net/zh_cn/ABOUT/) + +## 书籍 + +- [《深入理解BFE》](https://github.com/baidu/bfe-book) ：介绍网络接入的相关技术原理，说明BFE的设计思想，以及如何基于BFE搭建现代化的网络接入平台。现已开放全文阅读。 + +## 参与贡献 + +- 请首先在[issue列表](http://github.com/bfenetworks/bfe/issues)中创建一个issue +- 如有必要，请联系项目维护者/负责人进行进一步讨论 +- 请遵循golang编程规范 +- 详情请参阅[参与贡献指南](CONTRIBUTING.md) + +## 作者 +- 项目维护者: [MAINTAINERS](MAINTAINERS.md) +- 项目贡献者: [CONTRIBUTORS](CONTRIBUTORS.md) + +## 社区交流 +- [开源BFE用户论坛](https://github.com/bfenetworks/bfe/discussions) + +- **开源BFE微信公众号**：扫码关注公众号“BFE开源项目”，及时获取项目最新信息和技术分享 + + + + + +
+ +- **开源BFE用户微信群**：扫码加入，探讨和分享对BFE的建议、使用心得、疑问等 + + + + + +
+ +- **开源BFE开发者微信群**: [发送邮件](mailto:yangsijie@baidu.com)说明您的微信号及贡献(例如PR/Issue)，我们将及时邀请您加入 + +## 许可 +BFE基于Apache 2.0许可证，详见[LICENSE](LICENSE)文件说明 diff --git a/README.md b/README.md index 9ea865bd2..5b3ee97a6 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,8 @@ # BFE +English | [中文](README-CN.md) + + [![GitHub](https://img.shields.io/github/license/bfenetworks/bfe)](https://github.com/bfenetworks/bfe/blob/develop/LICENSE) [![Travis](https://img.shields.io/travis/com/bfenetworks/bfe)](https://travis-ci.com/bfenetworks/bfe) [![Go Report Card](https://goreportcard.com/badge/github.com/bfenetworks/bfe)](https://goreportcard.com/report/github.com/bfenetworks/bfe) @@ -40,7 +43,7 @@ BFE is a modern layer 7 load balancer from baidu. - Contributors: [CONTRIBUTORS](CONTRIBUTORS.md) ## Communication -- BFE community on Slack: [Sign up](https://join.slack.com/t/bfe-networks/shared_invite/zt-cn04xsqr-j7LDFmPkCuCZ39OLcHlMBA) and join channels on topics that interest you. +- BFE community on Slack: [Sign up](https://slack.cncf.io/) CNCF Slack and join bfe channel. - BFE developer group on WeChat: [Send a request mail](mailto:yangsijie@baidu.com) with your WeChat ID and a contribution you've made to BFE(such as a PR/Issue). We will invite you right away. ## License diff --git a/VERSION b/VERSION index 26aaba0e8..f0bb29e76 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.2.0 +1.3.0 diff --git a/bfe_config/bfe_conf/conf_basic.go b/bfe_config/bfe_conf/conf_basic.go index 7c3de643c..acfd94bcb 100644 --- a/bfe_config/bfe_conf/conf_basic.go +++ b/bfe_config/bfe_conf/conf_basic.go @@ -42,7 +42,7 @@ type ConfigBasic struct { HttpsPort int // listen port for https MonitorPort int // web server port for monitor MaxCpus int // number of max cpus to use - AcceptNum int // number of accept groutine for each listenr, default 1 + AcceptNum int // number of accept goroutine for each listener, default 1 // settings of layer-4 load balancer Layer4LoadBalancer string diff --git a/bfe_config/bfe_route_conf/route_rule_conf/basic_rule_tree.go b/bfe_config/bfe_route_conf/route_rule_conf/basic_rule_tree.go new file mode 100644 index 000000000..7847d7df6 --- /dev/null +++ b/bfe_config/bfe_route_conf/route_rule_conf/basic_rule_tree.go @@ -0,0 +1,218 @@ +// Copyright (c) 2019 The BFE Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// trees for matching basic route rule : (host+path) -> clusterName + +package route_rule_conf + +import ( + "fmt" + "strings" +) + +import ( + "github.com/armon/go-radix" +) + +import ( + "github.com/bfenetworks/bfe/bfe_util/string_reverse" +) + +const ( + treeMatchExact = iota // index to exact match tree + treeMatchWildcard // index to wildcard match tree + treeMatchTypeNum // index to exact match tree +) + +type hostTrees [treeMatchTypeNum]*radix.Tree // trees for host match +type pathTrees [treeMatchTypeNum]*radix.Tree // trees for path match + +// BasicRouteRuleTree implements radix trees for host+path matching +// hostTree (key:hostname, value:pathTree) +// pathTree (key:path, value:clusterName) +type BasicRouteRuleTree struct { + hosts hostTrees +} + +func NewBasicRouteRuleTree() *BasicRouteRuleTree { + return &BasicRouteRuleTree{ + hosts: [treeMatchTypeNum]*radix.Tree{radix.New(), radix.New()}, + } +} + +// Insert adds a new basic route rule into BasicRouteRuleTree +// key: hostname, value: pathTrees +func (r *BasicRouteRuleTree) Insert(ruleConf *BasicRouteRuleFile) error { + if len(ruleConf.Hostname) == 0 { + ruleConf.Hostname = append(ruleConf.Hostname, "*") + } + + if len(ruleConf.Path) == 0 { + ruleConf.Path = append(ruleConf.Path, "*") + } + + for _, host := range ruleConf.Hostname { + if host == "" { + // not allow + return fmt.Errorf("hostname is empty string") + } + + pathTree := r.hosts.insert(host) + for _, path := range ruleConf.Path { + if err := pathTree.insert(path, *ruleConf.ClusterName); err != nil { + return err + } + } + } + + return nil +} + +// insert adds a new node in hostTrees, key: hostname, value: pathTrees +// return node's value: pathTrees +func (ht *hostTrees) insert(host string) pathTrees { + var treeType int + var key string + + // remove * from wildcard host + // *.bar.foo.com -> .bar.foo.com + if host[0] == '*' { + key = host[1:] + treeType = treeMatchWildcard + } else { + key = host + treeType = treeMatchExact + } + + // call ReverseFqdnHost reverse hostname + // for case insensitive comparing, convert the reversed hostname to uppercase + // .bar.foo.com -> MOC.OOF.RAB. + key = strings.ToUpper(string_reverse.ReverseFqdnHost(key)) + + // return pathTree if already existed + if value, found := ht[treeType].Get(key); found { + return value.(pathTrees) + } + + // no host found, insert new node + value := pathTrees{radix.New(), radix.New()} + ht[treeType].Insert(key, value) + + return value +} + +// get returns pathTree for hostname +func (ht *hostTrees) get(host string) (pathTrees, bool) { + // convert key to uppercase for case insensitive comparing + // baz.bar.foo.com -> MOC.OOF.RAB.ZAB + key := strings.ToUpper(string_reverse.ReverseFqdnHost(host)) + + //exact match firstly + if value, found := ht[treeMatchExact].Get(key); found { + return value.(pathTrees), true + } + + // try wildcard match if exact match fail + // note: * only match one label in hostname. + // For example: *.aaa.com can match with bbb.aaa.com, but can't match with ccc.bbb.aaa.com + if matchedPrefix, value, found := ht[treeMatchWildcard].LongestPrefix(key); found { + + // get remaining(unmatched) part of a hostname without prefix + // For example: + // wildcard host *.bar.foo.com, key in tree would be MOC.OOF.RAB. + // to host baz.bar.foo.com, key for matching would be MOC.OOF.RAB.ZAB + // so, in this case, remainingPart = ZAB + remainingPart := strings.TrimPrefix(key, matchedPrefix) + + // the remaining part should not contain "." + // if the remaining part contain ".", it means '*' matches multiple labels in hostname, which is not allowed + if strings.Contains(remainingPart, ".") { + // not matched, try again to match empty string "", which match any hostname + if value, found := ht[treeMatchWildcard].Get(""); found { + // matched with "" + return value.(pathTrees), true + } + } else { + // matched with wildcard host + return value.(pathTrees), true + } + } + + return pathTrees{}, false +} + +// insert adds a new node in pathTree +// key : path, value: clusterName +func (pt *pathTrees) insert(path, cluster string) error { + var treeType int + var key string + + if len(path) == 0 { + return fmt.Errorf("empth path is not allowed") + } + + if path[len(path)-1] == '*' { + // wildcard path, remove trailing * + key = path[:len(path)-1] + + // append slash if no trailing one + // /foo, /foo/ or /foo/bar can match with /foo*, but /foobar can not + if len(key) > 0 && key[len(key)-1] != '/' { + key = key + "/" + } + treeType = treeMatchWildcard + } else { + key = path + treeType = treeMatchExact + } + + if old, updated := pt[treeType].Insert(key, cluster); updated { + // if key exist, return error + return fmt.Errorf("path[%s] is duplicated in same host, existing cluster: %s", path, old) + } + + return nil +} + +// get returns node's value (cluster name) referenced by a path +func (pt *pathTrees) get(path string) (string, bool) { + // exact match firstly + if value, found := pt[treeMatchExact].Get(path); found { + return value.(string), true + } + + // append trailing slash + if len(path) > 0 && path[len(path)-1] != '/' { + path = path + "/" + } + + // wildcard match + if _, value, found := pt[treeMatchWildcard].LongestPrefix(path); found { + return value.(string), true + } + + return "", false +} + +// Get returns cluster name by host and path +func (r *BasicRouteRuleTree) Get(host, path string) (string, bool) { + // match host to get pathTree + pathTree, found := r.hosts.get(host) + if !found { + return "", false + } + + // match path + return pathTree.get(path) +} diff --git a/bfe_config/bfe_route_conf/route_rule_conf/basic_rule_tree_test.go b/bfe_config/bfe_route_conf/route_rule_conf/basic_rule_tree_test.go new file mode 100644 index 000000000..9a16b2459 --- /dev/null +++ b/bfe_config/bfe_route_conf/route_rule_conf/basic_rule_tree_test.go @@ -0,0 +1,402 @@ +// Copyright (c) 2019 The BFE Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package route_rule_conf + +import ( + "testing" +) + +func TestHostMatchInBasicRuleTree(t *testing.T) { + + cluster := "c1" + rule := &BasicRouteRuleFile{ + Hostname: nil, + Path: nil, + ClusterName: &cluster, + } + + rule.Hostname = append(rule.Hostname, "") + rt := NewBasicRouteRuleTree() + if ret := rt.Insert(rule); ret == nil { + t.Errorf("insert empty host is not allowed") + } + + rule.Hostname = nil + rule.Hostname = append(rule.Hostname, "*") + rt.Insert(rule) + + c, ok := rt.Get("*", "/") + if !ok || c != cluster { + t.Errorf("should match") + } + + c, ok = rt.Get("any", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + + c, ok = rt.Get("bar.foo.com", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + + rule.Hostname = nil + rule.Hostname = append(rule.Hostname, "*.com") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + + c, ok = rt.Get("foo.com", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + _, ok = rt.Get("bar.foo.com", "/foo") + if ok { + t.Errorf("should not match") + } + + rule.Hostname = nil + rule.Hostname = append(rule.Hostname, "*.foo.com") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + + c, ok = rt.Get("bar.foo.com", "/") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/") + if ok { + t.Errorf("should not match *") + } + + rule.Hostname = nil + rule.Hostname = append(rule.Hostname, "foo.com") + rule.Hostname = append(rule.Hostname, "bar.com") + + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + c, ok = rt.Get("foo.com", "/") + if !ok || c != cluster { + t.Errorf("should match") + } + rt.Insert(rule) + c, ok = rt.Get("bar.com", "/") + if !ok || c != cluster { + t.Errorf("should match") + } + + rt = NewBasicRouteRuleTree() + rule.Hostname = nil + rule.Hostname = append(rule.Hostname, "foo.com") + cluster1 := "c1" + rule.ClusterName = &cluster1 + rt.Insert(rule) + + rule.Hostname = nil + rule.Hostname = append(rule.Hostname, "bar.com") + cluster2 := "c2" + rule.ClusterName = &cluster2 + rt.Insert(rule) + + c, ok = rt.Get("foo.com", "/") + if !ok || c != cluster1 { + t.Errorf("should match") + } + c, ok = rt.Get("bar.com", "/") + if !ok || c != cluster2 { + t.Errorf("should match") + } + + rule.Hostname = nil + rule.Hostname = append(rule.Hostname, "*.bar.com") + cluster3 := "c3" + rule.ClusterName = &cluster3 + rt.Insert(rule) + + c, ok = rt.Get("baz.bar.com", "/") + if !ok || c != cluster3 { + t.Errorf("should match") + } + +} + +func TestPathMatchInBasicRuleTree(t *testing.T) { + cluster := "c1" + rule := &BasicRouteRuleFile{ + Hostname: nil, + Path: nil, + ClusterName: &cluster, + } + + rule.Path = append(rule.Path, "") + rt := NewBasicRouteRuleTree() + if ret := rt.Insert(rule); ret == nil { + t.Errorf("empty path is not allowed") + } + + rule.Path = nil + rule.Path = append(rule.Path, "*") + rt.Insert(rule) + + c, ok := rt.Get("any", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + + c, ok = rt.Get("foo.com", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + + c, ok = rt.Get("foo.com", "foo") + if !ok || c != cluster { + t.Errorf("should match") + } + + c, ok = rt.Get("*", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + + c, ok = rt.Get("foo.com", "/") + if !ok || c != cluster { + t.Errorf("should match") + } + + c, ok = rt.Get("foo.com", "/foo") + if ok { + t.Errorf("should not match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/*") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + + c, ok = rt.Get("foo.com", "/") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/foo/bar") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "foo") + if ok { + t.Errorf("should not match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/foo") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + + c, ok = rt.Get("foo.com", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/bar") + if ok { + t.Errorf("should not match") + } + c, ok = rt.Get("foo.com", "/foobar") + if ok { + t.Errorf("should not match") + } + c, ok = rt.Get("foo.com", "/foo/") + if ok { + t.Errorf("should not match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/foo/") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + c, ok = rt.Get("foo.com", "/foo/") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/foo") + if ok { + t.Errorf("should not match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/foo*") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + c, ok = rt.Get("foo.com", "/foo/") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/foobar") + if ok { + t.Errorf("should not match") + } + c, ok = rt.Get("foo.com", "/bar") + if ok { + t.Errorf("should not match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/foo/*") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + c, ok = rt.Get("foo.com", "/foo/") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/foo/bar") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/foo") + if !ok || c != cluster { + t.Errorf("should match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/aaa/bb*") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + c, ok = rt.Get("foo.com", "/aaa/bbb") + if ok { + t.Errorf("should not match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/aaa/bbb*") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + c, ok = rt.Get("foo.com", "/aaa/bbb") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/aaa/bbb/") + if !ok || c != cluster { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/aaa/bbbxyz") + if ok { + t.Errorf("should not match") + } + c, ok = rt.Get("foo.com", "/aaa/bbb/xyz") + if !ok || c != cluster { + t.Errorf("should match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/aaa/bbb/*") + rt = NewBasicRouteRuleTree() + rt.Insert(rule) + c, ok = rt.Get("foo.com", "/aaa/bbb") + if !ok || c != cluster { + t.Errorf("should match") + } + + rt = NewBasicRouteRuleTree() + rule.Path = nil + rule.Path = append(rule.Path, "/*") + cluster1 := "c1" + rule.ClusterName = &cluster1 + rt.Insert(rule) + + rule.Path = nil + rule.Path = append(rule.Path, "/aaa*") + cluster2 := "c2" + rule.ClusterName = &cluster2 + rt.Insert(rule) + + c, ok = rt.Get("foo.com", "/aaa/ccc") + if !ok || c != cluster2 { + t.Errorf("should match") + } + + rule.Path = nil + rule.Path = append(rule.Path, "/aaa/bbb*") + cluster3 := "c3" + rule.ClusterName = &cluster3 + rt.Insert(rule) + + c, ok = rt.Get("foo.com", "/aaa/bbb") + if !ok || c != cluster3 { + t.Errorf("should match") + } + c, ok = rt.Get("foo.com", "/ccc") + if !ok || c != cluster1 { + t.Errorf("should match") + } + + rt = NewBasicRouteRuleTree() + rule.Path = nil + rule.Path = append(rule.Path, "/foo") + cluster1 = "c1" + rule.ClusterName = &cluster1 + rt.Insert(rule) + + rule.Path = nil + rule.Path = append(rule.Path, "/foo*") + cluster2 = "c2" + rule.ClusterName = &cluster2 + rt.Insert(rule) + + c, ok = rt.Get("foo.com", "/foo") + if !ok || c != cluster1 { + t.Errorf("should match") + } + +} + +func TestHostAndPathMatchInBasicRuleTree(t *testing.T) { + cluster := "c1" + rule := &BasicRouteRuleFile{ + Hostname: nil, + Path: nil, + ClusterName: &cluster, + } + + rule.Hostname = append(rule.Hostname, "aaa.foo.com") + rule.Hostname = append(rule.Hostname, "bbb.foo.com") + rule.Hostname = append(rule.Hostname, "ccc.foo.com") + rule.Path = append(rule.Path, "/aaa/bbb") + rule.Path = append(rule.Path, "/ccc/ddd") + rule.Path = append(rule.Path, "/eee/fff*") + cluster1 := "c1" + rule.ClusterName = &cluster1 + rt := NewBasicRouteRuleTree() + rt.Insert(rule) + + c, ok := rt.Get("aaa.foo.com", "/aaa/bbb") + if !ok || c != cluster1 { + t.Errorf("should match") + } + c, ok = rt.Get("ccc.foo.com", "/eee/fff/ggg") + if !ok || c != cluster1 { + t.Errorf("should match") + } + +} diff --git a/bfe_config/bfe_route_conf/route_rule_conf/route_table_load.go b/bfe_config/bfe_route_conf/route_rule_conf/route_table_load.go index b24168bea..c901fa850 100644 --- a/bfe_config/bfe_route_conf/route_rule_conf/route_table_load.go +++ b/bfe_config/bfe_route_conf/route_rule_conf/route_table_load.go @@ -20,6 +20,7 @@ import ( "errors" "fmt" "os" + "strings" ) import ( @@ -27,52 +28,106 @@ import ( "github.com/bfenetworks/bfe/bfe_util/json" ) -// RouteRule is composed by a condition and cluster to serve -type RouteRule struct { +const ( + AdvancedMode = "ADVANCED_MODE" +) + +// BasicRouteRule is for host+path routing +// [host, path] -> cluster +type BasicRouteRule struct { + Hostname []string + Path []string + ClusterName string +} + +// AdvancedRouteRule is composed by a condition and cluster to serve +type AdvancedRouteRule struct { Cond condition.Condition ClusterName string } -type RouteRuleFile struct { +type BasicRouteRuleFile struct { + Hostname []string + Path []string + ClusterName *string +} + +type AdvancedRouteRuleFile struct { Cond *string ClusterName *string } -// RouteRules is a list of rule. -type RouteRules []RouteRule -type RouteRuleFiles []RouteRuleFile +type BasicRouteRules []BasicRouteRule +type AdvancedRouteRules []AdvancedRouteRule + +type BasicRouteRuleFiles []BasicRouteRuleFile +type AdvancedRouteRuleFiles []AdvancedRouteRuleFile -// ProductRouteRule holds mapping from product to rules. -type ProductRouteRule map[string]RouteRules -type ProductRouteRuleFile map[string]RouteRuleFiles +type ProductBasicRouteRule map[string]BasicRouteRules +type ProductBasicRouteTree map[string]*BasicRouteRuleTree +type ProductAdvancedRouteRule map[string]AdvancedRouteRules + +type ProductAdvancedRouteRuleFile map[string]AdvancedRouteRuleFiles +type ProductBasicRouteRuleFile map[string]BasicRouteRuleFiles type RouteTableFile struct { - Version *string // version of the config - ProductRule *ProductRouteRuleFile // product => rules + Version *string // version of the config + + // product => rules (basic rule) + BasicRule *ProductBasicRouteRuleFile + + // product => rules (advanced rule) + ProductRule *ProductAdvancedRouteRuleFile } type RouteTableConf struct { - Version string // version of the config - RuleMap ProductRouteRule + Version string // version of the config + BasicRuleMap ProductBasicRouteRule + BasicRuleTree ProductBasicRouteTree + AdvancedRuleMap ProductAdvancedRouteRule } -func convert(fileConf *RouteTableFile) (*RouteTableConf, error) { - conf := &RouteTableConf{ - RuleMap: make(ProductRouteRule), - } - +func Convert(fileConf *RouteTableFile) (*RouteTableConf, error) { if fileConf.Version == nil { return nil, errors.New("no Version") } - if fileConf.ProductRule == nil { + if fileConf.BasicRule == nil && fileConf.ProductRule == nil { return nil, errors.New("no product rule") } + // convert basic rule + productBasicRules, productRuleTree, err := convertBasicRule(fileConf.BasicRule) + if err != nil { + return nil, err + } + + // convert advanced rule + productAdvancedRules, err := convertAdvancedRule(fileConf.ProductRule) + if err != nil { + return nil, err + } + + conf := &RouteTableConf{ + BasicRuleMap: productBasicRules, + BasicRuleTree: productRuleTree, + AdvancedRuleMap: productAdvancedRules, + } + conf.Version = *fileConf.Version - for product, ruleFiles := range *fileConf.ProductRule { - rules := make(RouteRules, len(ruleFiles)) + return conf, nil +} + +func convertAdvancedRule(ProductRule *ProductAdvancedRouteRuleFile) (ProductAdvancedRouteRule, error) { + productRules := make(ProductAdvancedRouteRule) + if ProductRule == nil { + return productRules, nil + } + + // convert advanced rule + for product, ruleFiles := range *ProductRule { + rules := make(AdvancedRouteRules, len(ruleFiles)) for i, ruleFile := range ruleFiles { if ruleFile.ClusterName == nil { return nil, errors.New("no cluster name") @@ -90,10 +145,94 @@ func convert(fileConf *RouteTableFile) (*RouteTableConf, error) { rules[i].Cond = cond } - conf.RuleMap[product] = rules + productRules[product] = rules } + return productRules, nil +} - return conf, nil +func convertBasicRule(ProductRule *ProductBasicRouteRuleFile) (ProductBasicRouteRule, ProductBasicRouteTree, error) { + productRuleMap := make(ProductBasicRouteRule) + productRuleTree := make(ProductBasicRouteTree) + + if ProductRule == nil { + return productRuleMap, productRuleTree, nil + } + + for product, ruleFiles := range *ProductRule { + ruleTrees := NewBasicRouteRuleTree() + ruleList := make(BasicRouteRules, len(ruleFiles)) + + for i, ruleFile := range ruleFiles { + + if ruleFile.ClusterName == nil { + return nil, nil, fmt.Errorf("no cluster name in basic route rule for (%s, %d)", product, i) + } + + if len(ruleFile.Hostname) == 0 && len(ruleFile.Path) == 0 { + return nil, nil, fmt.Errorf("no hostname or path in basic route rule for (%s, %d)", product, i) + } + + for _, host := range ruleFile.Hostname { + if err := checkHostInBasicRule(host); err != nil { + return nil, nil, fmt.Errorf("host[%s] is invalid for (%s, %d), err: %s ", host, product, i, err.Error()) + } + } + + for _, path := range ruleFile.Path { + if err := checkPathInBasicRule(path); err != nil { + return nil, nil, fmt.Errorf("path[%s] is invalid (%s, %d), err: %s ", path, product, i, err.Error()) + } + } + + ruleList[i].Hostname = ruleFile.Hostname + ruleList[i].Path = ruleFile.Path + ruleList[i].ClusterName = *ruleFile.ClusterName + + if err := ruleTrees.Insert(&ruleFile); err != nil { + return nil, nil, err + } + } + productRuleMap[product] = ruleList + productRuleTree[product] = ruleTrees + } + + return productRuleMap, productRuleTree, nil +} + +// checkHostInBasicRule verify host's wildcard pattern +// only one * is allowed, eg: *.foo.com +func checkHostInBasicRule(host string) error { + if host == "" { + return errors.New("hostname is nil or empty") + } + + if strings.Count(host, "*") > 1 { + return errors.New("only one * is allowed in a hostname") + } + + if strings.Count(host, "*") == 1 { + if host != "*" && !strings.HasPrefix(host, "*.") { + return errors.New("format error in wildcard host") + } + } + + return nil +} + +// checkPathInBasicRule verify path's wildcard pattern +// only one * at end of path is allowed +func checkPathInBasicRule(path string) error { + if path == "" { + return errors.New("path is nil or empty") + } + if strings.Count(path, "*") > 1 { + return errors.New("only one * is allowed in path") + } + + if strings.Count(path, "*") == 1 && path[len(path)-1] != '*' { + return errors.New("* must appear as last character of path") + } + return nil } func (conf *RouteTableConf) LoadAndCheck(filename string) (string, error) { @@ -112,7 +251,7 @@ func (conf *RouteTableConf) LoadAndCheck(filename string) (string, error) { return "", err } - pConf, err := convert(&fileConf) + pConf, err := Convert(&fileConf) if err != nil { return "", err } diff --git a/bfe_config/bfe_route_conf/route_rule_conf/route_table_load_test.go b/bfe_config/bfe_route_conf/route_rule_conf/route_table_load_test.go index b00450028..e82ffee71 100644 --- a/bfe_config/bfe_route_conf/route_rule_conf/route_table_load_test.go +++ b/bfe_config/bfe_route_conf/route_rule_conf/route_table_load_test.go @@ -26,9 +26,83 @@ func TestLoad(t *testing.T) { rt, err := RouteConfLoad(fn) if err != nil { t.Errorf("route conf load error %s", err) + return } - if len(rt.RuleMap["product-b"]) != 2 { - t.Errorf("product-2 condition len is not 2") + if len(rt.AdvancedRuleMap["product-b"]) != 2 { + t.Errorf("product-b condition len is not 2") + } +} +func TestLoad1(t *testing.T) { + pwd, _ := os.Getwd() + fn := fmt.Sprintf("%s/testdata/basic_route_rule1.data", pwd) + rt, err := RouteConfLoad(fn) + if err != nil { + t.Errorf("route conf load error %s", err) + return + } + + if len(rt.BasicRuleMap["example_product"]) != 3 { + t.Errorf("example_product rule number is not 3") + } +} +func TestLoad2(t *testing.T) { + pwd, _ := os.Getwd() + fn := fmt.Sprintf("%s/testdata/basic_route_rule2.data", pwd) + _, err := RouteConfLoad(fn) + if err == nil { + t.Errorf("route conf load should failed") + } + +} + +func TestLoad3(t *testing.T) { + pwd, _ := os.Getwd() + fn := fmt.Sprintf("%s/testdata/basic_route_rule3.data", pwd) + rt, err := RouteConfLoad(fn) + if err != nil { + t.Errorf("route conf load error %s", err) + return + } + + if len(rt.BasicRuleMap["example_product"]) != 3 { + t.Errorf("example_product len is not 3") + } + +} + +func TestLoad4(t *testing.T) { + pwd, _ := os.Getwd() + fn := fmt.Sprintf("%s/testdata/basic_route_rule4.data", pwd) + _, err := RouteConfLoad(fn) + if err == nil { + t.Errorf("route conf load should fail ") + } +} + +func TestLoad5(t *testing.T) { + pwd, _ := os.Getwd() + fn := fmt.Sprintf("%s/testdata/basic_route_rule5.data", pwd) + _, err := RouteConfLoad(fn) + if err == nil { + t.Errorf("route conf load should fail") + } +} + +func TestLoad6(t *testing.T) { + pwd, _ := os.Getwd() + fn := fmt.Sprintf("%s/testdata/basic_route_rule6.data", pwd) + _, err := RouteConfLoad(fn) + if err == nil { + t.Errorf("route conf load should fail") + } +} + +func TestLoad7(t *testing.T) { + pwd, _ := os.Getwd() + fn := fmt.Sprintf("%s/testdata/basic_route_rule7.data", pwd) + _, err := RouteConfLoad(fn) + if err == nil { + t.Errorf("route conf load should fail") } } diff --git a/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule1.data b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule1.data new file mode 100644 index 000000000..1bb18fdc6 --- /dev/null +++ b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule1.data @@ -0,0 +1,20 @@ +{ + "Version": "685", + "BasicRule":{ + "example_product" : [ + { + "Hostname": ["www.a.com","*"], + "Path":["/a","/b"], + "ClusterName": "cluster_example_1" + }, + { + "Hostname": ["*.foo.com","*.bar.com"], + "ClusterName": "cluster_example_2" + }, + { + "Path":["/a*","/b*"], + "ClusterName": "cluster_example_3" + } + ] + } +} diff --git a/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule2.data b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule2.data new file mode 100644 index 000000000..1b23979b9 --- /dev/null +++ b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule2.data @@ -0,0 +1,20 @@ +{ + "Version": "685", + "BasicRule":{ + "example_product" : [ + { + "ClusterName": "cluster_example_1" + }, + { + "Hostname": ["www.foo.com",""], + "Path":["/a","/b"], + "ClusterName": "cluster_example_2" + }, + { + "Hostname": ["","www.ddd.com"], + "Path":["/a","/b"], + "ClusterName": "ADVANCED_MODE" + } + ] + } +} diff --git a/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule3.data b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule3.data new file mode 100644 index 000000000..7d5a97e5a --- /dev/null +++ b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule3.data @@ -0,0 +1,46 @@ +{ + "Version": "685", + "ProductRule": { + "product-a": [ + { + "ClusterName": "cluster_a", + "Cond": "default_t()" + } + ], + "product-b": [ + { + "ClusterName": "cluster_b1", + "Cond": "req_header_key_in(\"X-Bd-Bwsc\")" + }, + { + "ClusterName": "cluster_b2", + "Cond": "default_t()" + } + ], + "example_product": [ + { + "ClusterName": "cluster_example", + "Cond": "default_t()" + } + ] + }, + "BasicRule":{ + "example_product" : [ + { + "Hostname": ["www.a.com","www.b.com"], + "Path":["/a","/b"], + "ClusterName": "cluster_example_1" + }, + { + "Hostname": ["*.foo.com","www.bar.com"], + "Path":["/a","/b"], + "ClusterName": "cluster_example_2" + }, + { + "Hostname": ["*.ccc.com","www.ddd.com"], + "Path":["/a","/b"], + "ClusterName": "ADVANCED_MODE" + } + ] + } +} diff --git a/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule4.data b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule4.data new file mode 100644 index 000000000..756ab4f50 --- /dev/null +++ b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule4.data @@ -0,0 +1,20 @@ +{ + "Version": "685", + "ProductRule": { + "example_product": [ + { + "ClusterName": "cluster_example", + "Cond": "default_t()" + } + ] + }, + "BasicRule":{ + "example_product" : [ + { + "Hostname": ["*abc.a.com","www.b.com"], + "Path":["/a","/b"], + "ClusterName": "cluster_example_1" + } + ] + } +} \ No newline at end of file diff --git a/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule5.data b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule5.data new file mode 100644 index 000000000..394bbd54b --- /dev/null +++ b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule5.data @@ -0,0 +1,20 @@ +{ + "Version": "685", + "ProductRule": { + "example_product": [ + { + "ClusterName": "cluster_example", + "Cond": "default_t()" + } + ] + }, + "BasicRule":{ + "example_product" : [ + { + "Hostname": ["abc*.a.com","www.b.com"], + "Path":["/a","/b"], + "ClusterName": "cluster_example_1" + } + ] + } +} \ No newline at end of file diff --git a/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule6.data b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule6.data new file mode 100644 index 000000000..d101552a2 --- /dev/null +++ b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule6.data @@ -0,0 +1,20 @@ +{ + "Version": "685", + "ProductRule": { + "example_product": [ + { + "ClusterName": "cluster_example", + "Cond": "default_t()" + } + ] + }, + "BasicRule":{ + "example_product" : [ + { + "Hostname": ["*.*.a.com","www.b.com"], + "Path":["/a","/b"], + "ClusterName": "cluster_example_1" + } + ] + } +} \ No newline at end of file diff --git a/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule7.data b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule7.data new file mode 100644 index 000000000..ce01568aa --- /dev/null +++ b/bfe_config/bfe_route_conf/route_rule_conf/testdata/basic_route_rule7.data @@ -0,0 +1,20 @@ +{ + "Version": "685", + "ProductRule": { + "example_product": [ + { + "ClusterName": "cluster_example", + "Cond": "default_t()" + } + ] + }, + "BasicRule":{ + "example_product" : [ + { + "Hostname": ["*.foo.com"], + "Path":["/a/*/*","/b"], + "ClusterName": "cluster_example_1" + } + ] + } +} \ No newline at end of file diff --git a/bfe_http/cookie_test.go b/bfe_http/cookie_test.go index c6ae7eb10..077c8949e 100644 --- a/bfe_http/cookie_test.go +++ b/bfe_http/cookie_test.go @@ -151,7 +151,7 @@ var addCookieTests = []struct { func TestAddCookie(t *testing.T) { for i, tt := range addCookieTests { - req, _ := NewRequest("GET", "http://example.com/", nil) + req, _ := NewRequest(MethodGet, "http://example.com/", nil) for _, c := range tt.Cookies { req.AddCookie(c) } diff --git a/bfe_http/method.go b/bfe_http/method.go new file mode 100644 index 000000000..0f45582be --- /dev/null +++ b/bfe_http/method.go @@ -0,0 +1,32 @@ +// Copyright (c) 2021 The BFE Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Copyright 2009 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package bfe_http + +//Common HTTP methods,Unless otherwise noted, these are defined in RFC 7231 section 4.3. +const ( + MethodGet = "GET" + MethodHead = "HEAD" + MethodPost = "POST" + MethodPut = "PUT" + MethodPatch = "PATCH" // RFC 5789 + MethodDelete = "DELETE" + MethodConnect = "CONNECT" + MethodOptions = "OPTIONS" + MethodTrace = "TRACE" +) diff --git a/bfe_http/readrequest_test.go b/bfe_http/readrequest_test.go index 9feb69fb9..aa62dc8fc 100644 --- a/bfe_http/readrequest_test.go +++ b/bfe_http/readrequest_test.go @@ -60,7 +60,7 @@ var reqTests = []reqTest{ "abcdef\n???", &Request{ - Method: "GET", + Method: MethodGet, URL: &url.URL{ Scheme: "http", Host: "www.techcrunch.com", @@ -108,7 +108,7 @@ var reqTests = []reqTest{ "Host: foo.com\r\n\r\n", &Request{ - Method: "GET", + Method: MethodGet, URL: &url.URL{ Path: "/", }, @@ -135,7 +135,7 @@ var reqTests = []reqTest{ "Host: test\r\n\r\n", &Request{ - Method: "GET", + Method: MethodGet, URL: &url.URL{ Path: "//user@host/is/actually/a/path/", }, @@ -186,7 +186,7 @@ var reqTests = []reqTest{ "Trailer-Key: Trailer-Value\r\n" + "\r\n", &Request{ - Method: "POST", + Method: MethodPost, URL: &url.URL{ Path: "/", }, @@ -213,7 +213,7 @@ var reqTests = []reqTest{ "CONNECT www.google.com:443 HTTP/1.1\r\n\r\n", &Request{ - Method: "CONNECT", + Method: MethodConnect, URL: &url.URL{ Host: "www.google.com:443", }, @@ -238,7 +238,7 @@ var reqTests = []reqTest{ "CONNECT 127.0.0.1:6060 HTTP/1.1\r\n\r\n", &Request{ - Method: "CONNECT", + Method: MethodConnect, URL: &url.URL{ Host: "127.0.0.1:6060", }, @@ -263,7 +263,7 @@ var reqTests = []reqTest{ "CONNECT /_goRPC_ HTTP/1.1\r\n\r\n", &Request{ - Method: "CONNECT", + Method: MethodConnect, URL: &url.URL{ Path: "/_goRPC_", }, @@ -312,7 +312,7 @@ var reqTests = []reqTest{ { "OPTIONS * HTTP/1.1\r\nServer: foo\r\n\r\n", &Request{ - Method: "OPTIONS", + Method: MethodOptions, URL: &url.URL{ Path: "*", }, diff --git a/bfe_http/request.go b/bfe_http/request.go index 35c82eb88..6ba213584 100644 --- a/bfe_http/request.go +++ b/bfe_http/request.go @@ -384,7 +384,7 @@ func (req *Request) write(w io.Writer, usingProxy bool, extraHeaders Header) err ruri := req.URL.RequestURI() if usingProxy && req.URL.Scheme != "" && req.URL.Opaque == "" { ruri = req.URL.Scheme + "://" + host + ruri - } else if req.Method == "CONNECT" && req.URL.Path == "" { + } else if req.Method == MethodConnect && req.URL.Path == "" { // CONNECT requests normally give just the host and port, not a full URL. ruri = host } else { @@ -654,7 +654,7 @@ func ReadRequest(b *bfe_bufio.Reader, maxUriBytes int) (req *Request, err error) // that starts with a slash. It can be parsed with the regular URL parser, // and the path will end up in req.URL.Path, where it needs to be in order for // RPC to work. - justAuthority := req.Method == "CONNECT" && !strings.HasPrefix(rawurl, "/") + justAuthority := req.Method == MethodConnect && !strings.HasPrefix(rawurl, "/") if justAuthority { rawurl = "http://" + rawurl } @@ -832,7 +832,7 @@ func parsePostForm(r *Request) (vs url.Values, err error) { func (r *Request) ParseForm() error { var err error if r.PostForm == nil { - if r.Method == "POST" || r.Method == "PUT" { + if r.Method == MethodPost || r.Method == MethodPut { r.PostForm, err = parsePostForm(r) } if r.PostForm == nil { diff --git a/bfe_http/request_test.go b/bfe_http/request_test.go index a1d77b474..0cb5b9660 100644 --- a/bfe_http/request_test.go +++ b/bfe_http/request_test.go @@ -37,7 +37,7 @@ import ( ) func TestQuery(t *testing.T) { - req := &Request{Method: "GET"} + req := &Request{Method: MethodGet} req.URL, _ = url.Parse("http://www.google.com/search?q=foo&q=bar") if q := req.FormValue("q"); q != "foo" { t.Errorf(`req.FormValue("q") = %q, want "foo"`, q) @@ -45,7 +45,7 @@ func TestQuery(t *testing.T) { } func TestPostQuery(t *testing.T) { - req, _ := NewRequest("POST", "http://www.google.com/search?q=foo&q=bar&both=x&prio=1&empty=not", + req, _ := NewRequest(MethodPost, "http://www.google.com/search?q=foo&q=bar&both=x&prio=1&empty=not", strings.NewReader("z=post&both=y&prio=2&empty=")) req.Header.Set("Content-Type", "application/x-www-form-urlencoded; param=value") @@ -92,7 +92,7 @@ var parseContentTypeTests = []parseContentTypeTest{ func TestParseFormUnknownContentType(t *testing.T) { for i, test := range parseContentTypeTests { req := &Request{ - Method: "POST", + Method: MethodPost, Header: Header(test.contentType), Body: ioutil.NopCloser(bytes.NewBufferString("body")), } @@ -107,10 +107,10 @@ func TestParseFormUnknownContentType(t *testing.T) { } func TestParseFormInitializeOnError(t *testing.T) { - nilBody, _ := NewRequest("POST", "http://www.google.com/search?q=foo", nil) + nilBody, _ := NewRequest(MethodPost, "http://www.google.com/search?q=foo", nil) tests := []*Request{ nilBody, - {Method: "GET", URL: nil}, + {Method: MethodGet, URL: nil}, } for i, req := range tests { err := req.ParseForm() @@ -125,7 +125,7 @@ func TestParseFormInitializeOnError(t *testing.T) { func TestMultipartReader(t *testing.T) { req := &Request{ - Method: "POST", + Method: MethodPost, Header: Header{"Content-Type": {`multipart/form-data; boundary="foo123"`}}, Body: ioutil.NopCloser(new(bytes.Buffer)), } @@ -142,7 +142,7 @@ func TestMultipartReader(t *testing.T) { } func TestSetBasicAuth(t *testing.T) { - r, _ := NewRequest("GET", "http://example.com/", nil) + r, _ := NewRequest(MethodGet, "http://example.com/", nil) r.SetBasicAuth("Aladdin", "open sesame") if g, e := r.Header.Get("Authorization"), "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ=="; g != e { t.Errorf("got header %q, want %q", g, e) @@ -180,7 +180,7 @@ func TestMultipartRequestAuto(t *testing.T) { func TestEmptyMultipartRequest(t *testing.T) { // Test that FormValue and FormFile automatically invoke // ParseMultipartForm and return the right values. - req, err := NewRequest("GET", "/", nil) + req, err := NewRequest(MethodGet, "/", nil) if err != nil { t.Errorf("NewRequest err = %q", err) } @@ -218,7 +218,7 @@ func TestReadRequestErrors(t *testing.T) { } func TestNewRequestHost(t *testing.T) { - req, err := NewRequest("GET", "http://localhost:1234/", nil) + req, err := NewRequest(MethodGet, "http://localhost:1234/", nil) if err != nil { t.Fatal(err) } @@ -246,7 +246,7 @@ func TestNewRequestContentLength(t *testing.T) { {readByte(io.NewSectionReader(strings.NewReader("xy"), 0, 6)), 0}, } for _, tt := range tests { - req, err := NewRequest("POST", "http://localhost/", tt.r) + req, err := NewRequest(MethodPost, "http://localhost/", tt.r) if err != nil { t.Fatal(err) } @@ -308,7 +308,7 @@ var getBasicAuthTests = []struct { func TestGetBasicAuth(t *testing.T) { for _, tt := range getBasicAuthTests { - r, _ := NewRequest("GET", "http://example.com/", nil) + r, _ := NewRequest(MethodGet, "http://example.com/", nil) r.SetBasicAuth(tt.username, tt.password) username, password, ok := r.BasicAuth() if ok != tt.ok || username != tt.username || password != tt.password { @@ -317,7 +317,7 @@ func TestGetBasicAuth(t *testing.T) { } } // Unauthenticated request. - r, _ := NewRequest("GET", "http://example.com/", nil) + r, _ := NewRequest(MethodGet, "http://example.com/", nil) username, password, ok := r.BasicAuth() if ok { t.Errorf("expected false from BasicAuth when the request is unauthenticated") @@ -350,7 +350,7 @@ var parseBasicAuthTests = []struct { func TestParseBasicAuth(t *testing.T) { for _, tt := range parseBasicAuthTests { - r, _ := NewRequest("GET", "http://example.com/", nil) + r, _ := NewRequest(MethodGet, "http://example.com/", nil) r.Header.Set("Authorization", tt.header) username, password, ok := r.BasicAuth() if ok != tt.ok || username != tt.username || password != tt.password { @@ -377,7 +377,7 @@ func (l logWrites) Write(p []byte) (n int, err error) { func TestRequestWriteBufferedWriter(t *testing.T) { got := []string{} - req, _ := NewRequest("GET", "http://foo.com/", nil) + req, _ := NewRequest(MethodGet, "http://foo.com/", nil) req.State = &RequestState{} req.Write(logWrites{t, &got}) want := []string{ @@ -406,7 +406,7 @@ func testMissingFile(t *testing.T, req *Request) { func newTestMultipartRequest(t *testing.T) *Request { b := bytes.NewBufferString(strings.Replace(message, "\n", "\r\n", -1)) - req, err := NewRequest("POST", "/", b) + req, err := NewRequest(MethodPost, "/", b) if err != nil { t.Fatal("NewRequest:", err) } diff --git a/bfe_http/requestwrite_test.go b/bfe_http/requestwrite_test.go index 428523466..5b79ba3ff 100644 --- a/bfe_http/requestwrite_test.go +++ b/bfe_http/requestwrite_test.go @@ -44,7 +44,7 @@ var reqWriteTests = []reqWriteTest{ // HTTP/1.1 => chunked coding; no body; no trailer { Req: Request{ - Method: "GET", + Method: MethodGet, URL: &url.URL{ Scheme: "http", Host: "www.techcrunch.com", @@ -91,7 +91,7 @@ var reqWriteTests = []reqWriteTest{ // HTTP/1.1 => chunked coding; body; empty trailer { Req: Request{ - Method: "GET", + Method: MethodGet, URL: &url.URL{ Scheme: "http", Host: "www.google.com", @@ -120,7 +120,7 @@ var reqWriteTests = []reqWriteTest{ // HTTP/1.1 POST => chunked coding; body; empty trailer { Req: Request{ - Method: "POST", + Method: MethodPost, URL: &url.URL{ Scheme: "http", Host: "www.google.com", @@ -153,7 +153,7 @@ var reqWriteTests = []reqWriteTest{ // HTTP/1.1 POST with Content-Length, no chunking { Req: Request{ - Method: "POST", + Method: MethodPost, URL: &url.URL{ Scheme: "http", Host: "www.google.com", @@ -188,7 +188,7 @@ var reqWriteTests = []reqWriteTest{ // HTTP/1.1 POST with Content-Length in headers { Req: Request{ - Method: "POST", + Method: MethodPost, URL: mustParseURL("http://example.com/"), Host: "example.com", Header: Header{ @@ -217,7 +217,7 @@ var reqWriteTests = []reqWriteTest{ // default to HTTP/1.1 { Req: Request{ - Method: "GET", + Method: MethodGet, URL: mustParseURL("/search"), Host: "www.google.com", }, @@ -231,7 +231,7 @@ var reqWriteTests = []reqWriteTest{ // Request with a 0 ContentLength and a 0 byte body. { Req: Request{ - Method: "POST", + Method: MethodPost, URL: mustParseURL("/"), Host: "example.com", ProtoMajor: 1, @@ -256,7 +256,7 @@ var reqWriteTests = []reqWriteTest{ // Request with a 0 ContentLength and a 1 byte body. { Req: Request{ - Method: "POST", + Method: MethodPost, URL: mustParseURL("/"), Host: "example.com", ProtoMajor: 1, @@ -282,7 +282,7 @@ var reqWriteTests = []reqWriteTest{ // Request with a ContentLength of 10 but a 5 byte body. { Req: Request{ - Method: "POST", + Method: MethodPost, URL: mustParseURL("/"), Host: "example.com", ProtoMajor: 1, @@ -296,7 +296,7 @@ var reqWriteTests = []reqWriteTest{ // Request with a ContentLength of 4 but an 8 byte body. { Req: Request{ - Method: "POST", + Method: MethodPost, URL: mustParseURL("/"), Host: "example.com", ProtoMajor: 1, @@ -310,7 +310,7 @@ var reqWriteTests = []reqWriteTest{ // Request with a 5 ContentLength and nil body. { Req: Request{ - Method: "POST", + Method: MethodPost, URL: mustParseURL("/"), Host: "example.com", ProtoMajor: 1, @@ -324,7 +324,7 @@ var reqWriteTests = []reqWriteTest{ // and doesn't add a User-Agent. { Req: Request{ - Method: "GET", + Method: MethodGet, URL: mustParseURL("/foo"), ProtoMajor: 1, ProtoMinor: 0, @@ -345,7 +345,7 @@ var reqWriteTests = []reqWriteTest{ // we don't change Go 1.0 behavior. { Req: Request{ - Method: "GET", + Method: MethodGet, Host: "", URL: &url.URL{ Scheme: "http", @@ -368,7 +368,7 @@ var reqWriteTests = []reqWriteTest{ // Opaque test #1 from golang.org/issue/4860 { Req: Request{ - Method: "GET", + Method: MethodGet, URL: &url.URL{ Scheme: "http", Host: "www.google.com", @@ -388,7 +388,7 @@ var reqWriteTests = []reqWriteTest{ // Opaque test #2 from golang.org/issue/4860 { Req: Request{ - Method: "GET", + Method: MethodGet, URL: &url.URL{ Scheme: "http", Host: "x.google.com", @@ -408,7 +408,7 @@ var reqWriteTests = []reqWriteTest{ // Testing custom case in header keys. Issue 5022. { Req: Request{ - Method: "GET", + Method: MethodGet, URL: &url.URL{ Scheme: "http", Host: "www.google.com", @@ -501,7 +501,7 @@ func (rc *closeChecker) Close() error { // inside a NopCloser, and that it serializes it correctly. func TestRequestWriteClosesBody(t *testing.T) { rc := &closeChecker{Reader: strings.NewReader("my body")} - req, _ := NewRequest("POST", "http://foo.com/", rc) + req, _ := NewRequest(MethodPost, "http://foo.com/", rc) req.State = &RequestState{} if req.ContentLength != 0 { t.Errorf("got req.ContentLength %d, want 0", req.ContentLength) diff --git a/bfe_http/response_test.go b/bfe_http/response_test.go index 43e951f1a..ccecce56e 100644 --- a/bfe_http/response_test.go +++ b/bfe_http/response_test.go @@ -59,7 +59,7 @@ var respTests = []respTest{ Proto: "HTTP/1.0", ProtoMajor: 1, ProtoMinor: 0, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{ "Connection": {"close"}, // TODO(rsc): Delete? }, @@ -84,7 +84,7 @@ var respTests = []respTest{ ProtoMajor: 1, ProtoMinor: 1, Header: Header{}, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Close: true, ContentLength: -1, }, @@ -105,7 +105,7 @@ var respTests = []respTest{ ProtoMajor: 1, ProtoMinor: 1, Header: Header{}, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Close: false, ContentLength: 0, }, @@ -127,7 +127,7 @@ var respTests = []respTest{ Proto: "HTTP/1.0", ProtoMajor: 1, ProtoMinor: 0, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{ "Connection": {"close"}, "Content-Length": {"10"}, @@ -157,7 +157,7 @@ var respTests = []respTest{ Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{}, Close: false, ContentLength: -1, @@ -184,7 +184,7 @@ var respTests = []respTest{ Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{}, Close: false, ContentLength: -1, @@ -206,7 +206,7 @@ var respTests = []respTest{ Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, - Request: dummyReq("HEAD"), + Request: dummyReq(MethodHead), Header: Header{}, TransferEncoding: []string{"chunked"}, Close: false, @@ -228,7 +228,7 @@ var respTests = []respTest{ Proto: "HTTP/1.0", ProtoMajor: 1, ProtoMinor: 0, - Request: dummyReq("HEAD"), + Request: dummyReq(MethodHead), Header: Header{"Content-Length": {"256"}}, TransferEncoding: nil, Close: true, @@ -250,7 +250,7 @@ var respTests = []respTest{ Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, - Request: dummyReq("HEAD"), + Request: dummyReq(MethodHead), Header: Header{"Content-Length": {"256"}}, TransferEncoding: nil, Close: false, @@ -271,7 +271,7 @@ var respTests = []respTest{ Proto: "HTTP/1.0", ProtoMajor: 1, ProtoMinor: 0, - Request: dummyReq("HEAD"), + Request: dummyReq(MethodHead), Header: Header{}, TransferEncoding: nil, Close: true, @@ -293,7 +293,7 @@ var respTests = []respTest{ Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{ "Content-Length": {"0"}, }, @@ -314,7 +314,7 @@ var respTests = []respTest{ Proto: "HTTP/1.0", ProtoMajor: 1, ProtoMinor: 0, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{}, Close: true, ContentLength: -1, @@ -333,7 +333,7 @@ var respTests = []respTest{ Proto: "HTTP/1.0", ProtoMajor: 1, ProtoMinor: 0, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{}, Close: true, ContentLength: -1, @@ -355,7 +355,7 @@ some body`, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{ "Content-Type": []string{"multipart/byteranges; boundary=18a75608c8f47cef"}, }, @@ -491,7 +491,7 @@ func TestReadResponseCloseInMiddle(t *testing.T) { buf.WriteString("Next Request Here") bufr := bfe_bufio.NewReader(&buf) - resp, err := ReadResponse(bufr, dummyReq("GET")) + resp, err := ReadResponse(bufr, dummyReq(MethodGet)) checkErr(err, "ReadResponse") expectedLength := int64(-1) if !test.chunked { @@ -612,7 +612,7 @@ func TestResponseContentLengthShortBody(t *testing.T) { "Content-Length: 123\r\n" + "\r\n" + shortBody)) - res, err := ReadResponse(br, &Request{Method: "GET"}) + res, err := ReadResponse(br, &Request{Method: MethodGet}) if err != nil { t.Fatal(err) } diff --git a/bfe_http/responsewrite_test.go b/bfe_http/responsewrite_test.go index 451f9099b..fa1d82cfd 100644 --- a/bfe_http/responsewrite_test.go +++ b/bfe_http/responsewrite_test.go @@ -37,7 +37,7 @@ func TestResponseWrite(t *testing.T) { StatusCode: 503, ProtoMajor: 1, ProtoMinor: 0, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{}, Body: ioutil.NopCloser(bytes.NewBufferString("abcdef")), ContentLength: 6, @@ -53,7 +53,7 @@ func TestResponseWrite(t *testing.T) { StatusCode: 200, ProtoMajor: 1, ProtoMinor: 0, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{}, Body: ioutil.NopCloser(bytes.NewBufferString("abcdef")), ContentLength: -1, @@ -68,7 +68,7 @@ func TestResponseWrite(t *testing.T) { StatusCode: 200, ProtoMajor: 1, ProtoMinor: 1, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{}, Body: ioutil.NopCloser(bytes.NewBufferString("abcdef")), ContentLength: 6, @@ -89,7 +89,7 @@ func TestResponseWrite(t *testing.T) { StatusCode: 204, ProtoMajor: 1, ProtoMinor: 1, - Request: dummyReq("GET"), + Request: dummyReq(MethodGet), Header: Header{ "Foo": []string{" Bar\nBaz "}, }, diff --git a/bfe_http/transfer.go b/bfe_http/transfer.go index 6c75911f7..2f5f1c883 100644 --- a/bfe_http/transfer.go +++ b/bfe_http/transfer.go @@ -42,8 +42,8 @@ type transferWriter struct { Body io.Reader BodyCloser io.Closer ResponseToHEAD bool - ContentLength int64 // -1 means unknown, 0 means exactly none Close bool + ContentLength int64 // -1 means unknown, 0 means exactly none TransferEncoding []string Header Header Trailer Header @@ -131,7 +131,7 @@ func newTransferWriter(r interface{}) (t *transferWriter, err error) { } func noBodyExpected(requestMethod string) bool { - return requestMethod == "HEAD" + return requestMethod == MethodHead } func (t *transferWriter) shouldSendContentLength() bool { @@ -262,11 +262,11 @@ type transferReader struct { // permits a body. See RFC2616, section 4.4. func bodyAllowedForStatus(status int) bool { switch { - case status >= 100 && status <= 199: + case status >= StatusContinue && status <= 199: return false - case status == 204: + case status == StatusNoContent: return false - case status == 304: + case status == StatusNotModified: return false } return true @@ -274,7 +274,7 @@ func bodyAllowedForStatus(status int) bool { // msg is *Request or *Response. func readTransfer(msg interface{}, r *bfe_bufio.Reader) (err error) { - t := &transferReader{RequestMethod: "GET"} + t := &transferReader{RequestMethod: MethodGet} // Unify input isResponse := false @@ -295,7 +295,7 @@ func readTransfer(msg interface{}, r *bfe_bufio.Reader) (err error) { t.ProtoMinor = rr.ProtoMinor // Transfer semantics for Requests are exactly like those for // Responses with status code 200, responding to a GET method - t.StatusCode = 200 + t.StatusCode = StatusOK default: panic("unexpected type") } @@ -315,7 +315,7 @@ func readTransfer(msg interface{}, r *bfe_bufio.Reader) (err error) { if err != nil { return err } - if isResponse && t.RequestMethod == "HEAD" { + if isResponse && t.RequestMethod == MethodHead { if n, err := parseContentLength(t.Header.GetDirect("Content-Length")); err != nil { return err } else { @@ -448,7 +448,7 @@ func fixLength(isResponse bool, status int, requestMethod string, header Header, return 0, nil } switch status { - case 204, 304: + case StatusNoContent, StatusNotModified: return 0, nil } @@ -469,7 +469,7 @@ func fixLength(isResponse bool, status int, requestMethod string, header Header, header.Del("Content-Length") } - if !isResponse && requestMethod == "GET" { + if !isResponse && requestMethod == MethodGet { // RFC 2616 doesn't explicitly permit nor forbid an // entity-body on a GET request so we permit one if // declared, but we default to 0 here (not -1 below) diff --git a/bfe_http/transport.go b/bfe_http/transport.go index 3f129d176..84091b63e 100644 --- a/bfe_http/transport.go +++ b/bfe_http/transport.go @@ -561,7 +561,7 @@ func (t *Transport) dialConn(cm *connectMethod) (*persistConn, error) { } case cm.targetScheme == "https": connectReq := &Request{ - Method: "CONNECT", + Method: MethodConnect, URL: &url.URL{Opaque: cm.targetAddr}, Host: cm.targetAddr, Header: make(Header), @@ -820,7 +820,7 @@ func (pc *persistConn) readLoop() { resp, err = ReadResponse(pc.br, rc.req) } } - hasBody := resp != nil && rc.req.Method != "HEAD" && resp.ContentLength != 0 + hasBody := resp != nil && rc.req.Method != MethodHead && resp.ContentLength != 0 if err != nil { pc.close() @@ -977,7 +977,7 @@ func (pc *persistConn) roundTrip(req *transportRequest) (resp *Response, err err // uncompress the gzip stream if we were the layer that // requested it. requestedGzip := false - if !pc.t.DisableCompression && req.Header.Get("Accept-Encoding") == "" && req.Method != "HEAD" { + if !pc.t.DisableCompression && req.Header.Get("Accept-Encoding") == "" && req.Method != MethodHead { // Request gzip only, not deflate. Deflate is ambiguous and // not as universally supported anyway. // See: http://www.gzip.org/zlib/zlib_faq.html#faq38 diff --git a/bfe_module/bfe_callback.go b/bfe_module/bfe_callback.go index 0cf1734c9..65fe108ee 100644 --- a/bfe_module/bfe_callback.go +++ b/bfe_module/bfe_callback.go @@ -30,15 +30,15 @@ import ( // Callback point. const ( - HandleAccept = 0 - HandleHandshake = 1 - HandleBeforeLocation = 2 - HandleFoundProduct = 3 - HandleAfterLocation = 4 - HandleForward = 5 - HandleReadResponse = 6 - HandleRequestFinish = 7 - HandleFinish = 8 + HandleAccept = iota + HandleHandshake + HandleBeforeLocation + HandleFoundProduct + HandleAfterLocation + HandleForward + HandleReadResponse + HandleRequestFinish + HandleFinish ) func CallbackPointName(point int) string { @@ -137,7 +137,7 @@ func (bcb *BfeCallbacks) GetHandlerList(point int) *HandlerList { return hl } -// ModuleHandlersGetJSON get info of hanlders +// ModuleHandlersGetJSON get info of handlers func (bcb *BfeCallbacks) ModuleHandlersGetJSON() ([]byte, error) { cbs := make(map[string][]string) diff --git a/bfe_module/bfe_callback_test.go b/bfe_module/bfe_callback_test.go new file mode 100644 index 000000000..fecbcecb5 --- /dev/null +++ b/bfe_module/bfe_callback_test.go @@ -0,0 +1,110 @@ +// Copyright (c) 2019 The BFE Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package bfe_module + +import ( + "testing" +) + +import ( + "github.com/stretchr/testify/assert" +) + +import ( + "github.com/bfenetworks/bfe/bfe_basic" + "github.com/bfenetworks/bfe/bfe_http" +) + +func TestCallbackPointName(t *testing.T) { + point2Name := map[int]string{ + HandleAccept: "HandleAccept", + HandleHandshake: "HandleHandshake", + HandleBeforeLocation: "HandleBeforeLocation", + HandleFoundProduct: "HandleFoundProduct", + HandleAfterLocation: "HandleAfterLocation", + HandleForward: "HandleForward", + HandleReadResponse: "HandleReadResponse", + HandleRequestFinish: "HandleRequestFinish", + HandleFinish: "HandleFinish", + -1: "HandleUnknown", + } + for key, value := range point2Name { + if e := CallbackPointName(key); e != value { + t.Errorf("TestCallbackPointName, expecting:\n%s\nGot:\n%s\n", value, e) + } + } +} + +func TestNewBfeCallbacks(t *testing.T) { + bcb := NewBfeCallbacks() + assert.Len(t, bcb.callbacks, 9) +} + +func TestBfeCallbacksAddFilter(t *testing.T) { + bcb := NewBfeCallbacks() + var err error + assert.NotNil(t, bcb) + err = bcb.AddFilter(HandleAccept, func(session *bfe_basic.Session) int { + return 0 + }) + assert.NoError(t, err) + + err = bcb.AddFilter(HandleBeforeLocation, func(req *bfe_basic.Request) (int, *bfe_http.Response) { + return 0, nil + }) + assert.NoError(t, err) + + err = bcb.AddFilter(HandleForward, func(req *bfe_basic.Request) int { + return 0 + }) + assert.NoError(t, err) + + err = bcb.AddFilter(HandleReadResponse, func(req *bfe_basic.Request, res *bfe_http.Response) int { + return 0 + }) + assert.NoError(t, err) + + err = bcb.AddFilter(HandleFinish, func(session *bfe_basic.Session) int { + return 0 + }) + assert.NoError(t, err) + + err = bcb.AddFilter(-1, func() {}) + assert.Error(t, err) + + bcb = &BfeCallbacks{} + bcb.callbacks = make(map[int]*HandlerList) + bcb.callbacks[HandleAccept] = NewHandlerList(-1) + err = bcb.AddFilter(HandleAccept, func(session *bfe_basic.Session) int { + return 0 + }) + assert.Error(t, err) + +} + +func TestBfeCallbacksGetHandlerList(t *testing.T) { + bcb := NewBfeCallbacks() + assert.NotNil(t, bcb) + assert.Nil(t, bcb.GetHandlerList(-1)) + assert.NotNil(t, bcb.GetHandlerList(HandleAccept)) + +} + +func TestBfeCallbacksModuleHandlersGetJSON(t *testing.T) { + bcb := NewBfeCallbacks() + assert.NotNil(t, bcb) + _, err := bcb.ModuleHandlersGetJSON() + assert.NoError(t, err) +} diff --git a/bfe_module/bfe_filter.go b/bfe_module/bfe_filter.go index e05b009ab..7cbe700d9 100644 --- a/bfe_module/bfe_filter.go +++ b/bfe_module/bfe_filter.go @@ -53,7 +53,7 @@ import ( "github.com/bfenetworks/bfe/bfe_http" ) -// RequestFilter filters incomming requests and return a response or nil. +// RequestFilter filters incoming requests and return a response or nil. // Filters are chained together into a HandlerList. type RequestFilter interface { FilterRequest(request *bfe_basic.Request) (int, *bfe_http.Response) diff --git a/bfe_module/bfe_handler_list.go b/bfe_module/bfe_handler_list.go index e4ea6a1b9..cbf7bc235 100644 --- a/bfe_module/bfe_handler_list.go +++ b/bfe_module/bfe_handler_list.go @@ -32,20 +32,30 @@ import ( // HandlerList type. const ( - HandlersAccept = 0 // for AcceptFilter - HandlersRequest = 1 // for RequestFilter - HandlersForward = 2 // for ForwardFilter - HandlersResponse = 3 // for ResponseFilter - HandlersFinish = 4 // for FinishFilter + // for AcceptFilter + HandlersAccept = iota + // for RequestFilter + HandlersRequest + // for ForwardFilter + HandlersForward + // for ResponseFilter + HandlersResponse + // for FinishFilter + HandlersFinish ) // Return value of handler. const ( - BfeHandlerFinish = 0 // to close the connection after response - BfeHandlerGoOn = 1 // to go on next handler - BfeHandlerRedirect = 2 // to redirect - BfeHandlerResponse = 3 // to send response - BfeHandlerClose = 4 // to close the connection directly, with no data sent. + // to close the connection after response + BfeHandlerFinish = iota + // to go on next handler + BfeHandlerGoOn + // to redirect + BfeHandlerRedirect + // to send response + BfeHandlerResponse + // to close the connection directly, with no data sent. + BfeHandlerClose ) type HandlerList struct { diff --git a/bfe_module/bfe_handler_list_test.go b/bfe_module/bfe_handler_list_test.go new file mode 100644 index 000000000..7b2d25801 --- /dev/null +++ b/bfe_module/bfe_handler_list_test.go @@ -0,0 +1,124 @@ +// Copyright (c) 2019 The BFE Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package bfe_module + +import ( + "testing" +) + +import ( + "github.com/stretchr/testify/assert" +) + +import ( + "github.com/bfenetworks/bfe/bfe_basic" + "github.com/bfenetworks/bfe/bfe_http" +) + +func TestHandlerListFilterAccept(t *testing.T) { + hl := NewHandlerList(HandlersAccept) + assert.NotNil(t, hl) + var err error + var session *bfe_basic.Session + err = hl.AddAcceptFilter(func(session *bfe_basic.Session) (int, error) { + return BfeHandlerGoOn, nil + }) + assert.Error(t, err) + + err = hl.AddAcceptFilter(func(session *bfe_basic.Session) int { + return BfeHandlerGoOn + }) + assert.NoError(t, err) + + retVal := hl.FilterAccept(session) + assert.EqualValues(t, BfeHandlerGoOn, retVal) +} + +func TestHandlerListFilterRequest(t *testing.T) { + hl := NewHandlerList(HandlersRequest) + assert.NotNil(t, hl) + var err error + var req *bfe_basic.Request + err = hl.AddRequestFilter(func(req *bfe_basic.Request) *bfe_http.Response { + return nil + }) + assert.Error(t, err) + + err = hl.AddRequestFilter(func(req *bfe_basic.Request) (int, *bfe_http.Response) { + return BfeHandlerGoOn, nil + }) + assert.NoError(t, err) + + retVal, _ := hl.FilterRequest(req) + assert.EqualValues(t, BfeHandlerGoOn, retVal) +} + +func TestHandlerListFilterForward(t *testing.T) { + hl := NewHandlerList(HandlersForward) + assert.NotNil(t, hl) + var err error + var req *bfe_basic.Request + err = hl.AddForwardFilter(func(req *bfe_basic.Request) (int, error) { + return BfeHandlerRedirect, nil + }) + assert.Error(t, err) + + err = hl.AddForwardFilter(func(req *bfe_basic.Request) int { + return BfeHandlerRedirect + }) + assert.NoError(t, err) + + retVal := hl.FilterForward(req) + assert.EqualValues(t, BfeHandlerRedirect, retVal) +} + +func TestHandlerListFilterResponse(t *testing.T) { + hl := NewHandlerList(HandlersResponse) + assert.NotNil(t, hl) + var err error + var req *bfe_basic.Request + var res *bfe_http.Response + err = hl.AddResponseFilter(func(req *bfe_basic.Request) int { + return BfeHandlerResponse + }) + assert.Error(t, err) + + err = hl.AddResponseFilter(func(req *bfe_basic.Request, res *bfe_http.Response) int { + return BfeHandlerResponse + }) + assert.NoError(t, err) + + retVal := hl.FilterResponse(req, res) + assert.EqualValues(t, BfeHandlerResponse, retVal) +} + +func TestHandlerListFilterFinish(t *testing.T) { + hl := NewHandlerList(HandlersFinish) + assert.NotNil(t, hl) + var err error + var session *bfe_basic.Session + err = hl.AddFinishFilter(func(session *bfe_basic.Session) (int, error) { + return BfeHandlerFinish, nil + }) + assert.Error(t, err) + + err = hl.AddFinishFilter(func(session *bfe_basic.Session) int { + return BfeHandlerFinish + }) + assert.NoError(t, err) + + retVal := hl.FilterFinish(session) + assert.EqualValues(t, BfeHandlerFinish, retVal) +} diff --git a/bfe_module/bfe_module_test.go b/bfe_module/bfe_module_test.go new file mode 100644 index 000000000..63ce29566 --- /dev/null +++ b/bfe_module/bfe_module_test.go @@ -0,0 +1,93 @@ +// Copyright (c) 2021 The BFE Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package bfe_module + +import ( + "github.com/baidu/go-lib/web-monitor/web_monitor" + "github.com/stretchr/testify/assert" + "testing" +) + +//mock module +type testModule struct { +} + +func (tm testModule) Name() string { + return "tm" +} + +func (tm testModule) Init(cbs *BfeCallbacks, whs *web_monitor.WebHandlers, cr string) error { + return nil +} + +var tm testModule + +func init() { + AddModule(tm) +} + +func TestNewBfeModules(t *testing.T) { + bms := NewBfeModules() + assert.NotNil(t, bms) +} + +func TestBfeModulesRegisterModule(t *testing.T) { + bms := NewBfeModules() + assert.NotNil(t, bms) + var err error + err = bms.RegisterModule("") + assert.Error(t, err) + + err = bms.RegisterModule("tm") + assert.NoError(t, err) +} + +func TestBfeModulesGetModule(t *testing.T) { + bms := NewBfeModules() + assert.NotNil(t, bms) + err := bms.RegisterModule("tm") + assert.NoError(t, err) + + bm := bms.GetModule("tm") + assert.NotNil(t, bm) +} + +func TestBfeModulesInit(t *testing.T) { + bms := NewBfeModules() + assert.NotNil(t, bms) + var err error + err = bms.RegisterModule("tm") + assert.Nil(t, err) + + err = bms.Init(nil, nil, "") + assert.Nil(t, err) +} + +func TestModConfPath(t *testing.T) { + s := ModConfPath("/home/bfe/conf", "mod_access") + assert.EqualValues(t, "/home/bfe/conf/mod_access/mod_access.conf", s) +} + +func TestModConfDir(t *testing.T) { + s := ModConfDir("/home/bfe/conf", "mod_access") + assert.EqualValues(t, "/home/bfe/conf/mod_access", s) +} + +func TestModuleStatusGetJSON(t *testing.T) { + var err error + TestBfeModulesInit(t) + _, err = ModuleStatusGetJSON() + assert.NoError(t, err) +} diff --git a/bfe_modules/mod_auth_jwt/auth_jwt_rule_load.go b/bfe_modules/mod_auth_jwt/auth_jwt_rule_load.go index a187fca12..5789c1ef9 100644 --- a/bfe_modules/mod_auth_jwt/auth_jwt_rule_load.go +++ b/bfe_modules/mod_auth_jwt/auth_jwt_rule_load.go @@ -21,7 +21,7 @@ import ( ) import ( - jwt "github.com/dgrijalva/jwt-go" + "github.com/golang-jwt/jwt" jose "gopkg.in/square/go-jose.v2" ) diff --git a/bfe_modules/mod_auth_jwt/mod_auth_jwt.go b/bfe_modules/mod_auth_jwt/mod_auth_jwt.go index a4fb42008..a69a66750 100644 --- a/bfe_modules/mod_auth_jwt/mod_auth_jwt.go +++ b/bfe_modules/mod_auth_jwt/mod_auth_jwt.go @@ -24,7 +24,7 @@ import ( "github.com/baidu/go-lib/log" "github.com/baidu/go-lib/web-monitor/metrics" "github.com/baidu/go-lib/web-monitor/web_monitor" - jwt "github.com/dgrijalva/jwt-go" + "github.com/golang-jwt/jwt" ) import ( diff --git a/bfe_proxy/conn.go b/bfe_proxy/conn.go index 6e194b5c3..ef415cd6b 100644 --- a/bfe_proxy/conn.go +++ b/bfe_proxy/conn.go @@ -46,7 +46,7 @@ const ( // "The sender must ensure that all the protocol header is sent at once. This block // is always smaller than an MSS, so there is no reason for it to be segmented at // the beginning of the connection." - defualtMaxProxyHeaderBytes int64 = 2048 + defaultMaxProxyHeaderBytes int64 = 2048 // noLimit is an effective infinite upper bound for io.LimitedReader noLimit int64 = (1 << 63) - 1 @@ -74,7 +74,7 @@ func NewConn(conn net.Conn, headerTimeout time.Duration, maxProxyHeaderBytes int headerTimeout = defaultProxyHeaderTimeout } if maxProxyHeaderBytes <= 0 { - maxProxyHeaderBytes = defualtMaxProxyHeaderBytes + maxProxyHeaderBytes = defaultMaxProxyHeaderBytes } pConn := new(Conn) @@ -167,7 +167,6 @@ func (p *Conn) checkProxyHeaderOnce() { p.once.Do(func() { if err := p.checkProxyHeader(); err != nil { log.Logger.Error("bfe_proxy: Failed to read proxy header: %v", err) - p.Close() } }) } @@ -188,7 +187,7 @@ func (p *Conn) checkProxyHeader() error { return nil } if err != nil { - p.conn.Close() + p.Close() p.headerErr = err return err } @@ -197,14 +196,14 @@ func (p *Conn) checkProxyHeader() error { srcAddr := net.JoinHostPort(hdr.SourceAddress.String(), fmt.Sprintf("%d", hdr.SourcePort)) p.srcAddr, err = net.ResolveTCPAddr(hdr.TransportProtocol.String(), srcAddr) if err != nil { /* never go here */ - p.conn.Close() + p.Close() return err } dstAddr := net.JoinHostPort(hdr.DestinationAddress.String(), fmt.Sprintf("%d", hdr.DestinationPort)) p.dstAddr, err = net.ResolveTCPAddr(hdr.TransportProtocol.String(), dstAddr) if err != nil { /* never go here */ - p.conn.Close() + p.Close() return err } diff --git a/bfe_proxy/conn_test.go b/bfe_proxy/conn_test.go index e6c5f3c5e..14d63f1f3 100644 --- a/bfe_proxy/conn_test.go +++ b/bfe_proxy/conn_test.go @@ -241,7 +241,7 @@ func checkAddrEqual(a, b net.Addr) bool { func checkError(t *testing.T, errWant string, errGot error) { if len(errWant) == 0 && errGot != nil { - t.Errorf("Unexpect error: %v", errGot) + t.Errorf("Unexpected error: %v", errGot) return } if len(errWant) > 0 && errGot == nil { diff --git a/bfe_proxy/v2.go b/bfe_proxy/v2.go index 282753559..700299960 100644 --- a/bfe_proxy/v2.go +++ b/bfe_proxy/v2.go @@ -82,7 +82,7 @@ type _addr6 struct { // // Note: binary header format: // - Signature [1~12] -// - Version and Commannd [13] +// - Version and Command [13] // - Protocol and Address Family [14] // - Address Length [15] // - Additional TLVs [optional] diff --git a/bfe_route/host_table.go b/bfe_route/host_table.go index 3a1820840..bba06bf66 100644 --- a/bfe_route/host_table.go +++ b/bfe_route/host_table.go @@ -27,6 +27,7 @@ import ( "github.com/bfenetworks/bfe/bfe_config/bfe_route_conf/route_rule_conf" "github.com/bfenetworks/bfe/bfe_config/bfe_route_conf/vip_rule_conf" "github.com/bfenetworks/bfe/bfe_route/trie" + "github.com/bfenetworks/bfe/bfe_util/string_reverse" ) var ( @@ -42,11 +43,15 @@ type HostTable struct { hostTable host_rule_conf.Host2HostTag // for get host-tag hostTagTable host_rule_conf.HostTag2Product // for get product name by hostname - vipTable vip_rule_conf.Vip2Product // for get proudct name by vip (backup) + vipTable vip_rule_conf.Vip2Product // for get product name by vip (backup) defaultProduct string // default product name - hostTrie *trie.Trie - productRouteTable route_rule_conf.ProductRouteRule // all product's route rules + hostTrie *trie.Trie + + productBasicRouteTable route_rule_conf.ProductBasicRouteRule // all product's basic route rules list + productBasicRouteTree route_rule_conf.ProductBasicRouteTree // all product's basic route rules tree + productAdvancedRouteTable route_rule_conf.ProductAdvancedRouteRule // all product's advanced route rules + } type Versions struct { @@ -56,10 +61,11 @@ type Versions struct { } type Status struct { - HostTableSize int - HostTagTableSize int - VipTableSize int - ProductRouteTableSize int + HostTableSize int + HostTagTableSize int + VipTableSize int + ProductRouteTableSize int + ProductBasicRouteTableSize int } type route struct { @@ -90,7 +96,9 @@ func (t *HostTable) updateVipTable(conf vip_rule_conf.VipConf) { // updateRouteTable updates product Route Rule func (t *HostTable) updateRouteTable(conf *route_rule_conf.RouteTableConf) { t.versions.ProductRoute = conf.Version - t.productRouteTable = conf.RuleMap + t.productBasicRouteTree = conf.BasicRuleTree + t.productBasicRouteTable = conf.BasicRuleMap + t.productAdvancedRouteTable = conf.AdvancedRuleMap } // Update updates host table @@ -133,8 +141,26 @@ func (t *HostTable) LookupHostTagAndProduct(req *bfe_basic.Request) error { func (t *HostTable) LookupCluster(req *bfe_basic.Request) error { var clusterName string - // get route rules - rules, ok := t.productRouteTable[req.Route.Product] + // match basic route rules + basicRules, ok := t.productBasicRouteTree[req.Route.Product] + if ok { + host := strings.SplitN(req.HttpRequest.Host, ":", 2)[0] + + path := "" + if req.HttpRequest.URL != nil { + path = req.HttpRequest.URL.Path + } + + clusterName, found := basicRules.Get(host, path) + if found && clusterName != route_rule_conf.AdvancedMode { + // set clusterName + req.Route.ClusterName = clusterName + return nil + } + } + + // match advanced route rules + rules, ok := t.productAdvancedRouteTable[req.Route.Product] if !ok { req.Route.ClusterName = "" req.Route.Error = ErrNoProductRule @@ -215,7 +241,8 @@ func (t *HostTable) GetVersions() Versions { // GetStatus return status of host table. func (t *HostTable) GetStatus() Status { var s Status - s.ProductRouteTableSize = len(t.productRouteTable) + s.ProductBasicRouteTableSize = len(t.productBasicRouteTable) + s.ProductRouteTableSize = len(t.productAdvancedRouteTable) s.HostTableSize = len(t.hostTable) s.HostTagTableSize = len(t.hostTagTable) s.VipTableSize = len(t.vipTable) @@ -229,7 +256,7 @@ func (t *HostTable) findHostRoute(host string) (route, error) { host = strings.ToLower(host) // get host-tag by hostname - match, ok := t.hostTrie.Get(strings.Split(reverseFqdnHost(hostnameStrip(host)), ".")) + match, ok := t.hostTrie.Get(strings.Split(string_reverse.ReverseFqdnHost(hostnameStrip(host)), ".")) if ok { // get route success, return return match.(route), nil @@ -255,28 +282,13 @@ func hostnameStrip(hostname string) string { return strings.Split(hostname, ":")[0] } -// reverseFqdnHost reverse host to make prefix tree smaller. -// i.e.: www.baidu.com news.baidu.com -> moc.udiab.swen moc.udiab.www will have same prefix -func reverseFqdnHost(host string) string { - r := []rune(host) - for i, j := 0, len(r)-1; i < j; i, j = i+1, j-1 { - r[i], r[j] = r[j], r[i] - } - - if len(r) > 0 && r[0] == '.' { - r = r[1:] - } - - return string(r) -} - func buildHostRoute(conf host_rule_conf.HostConf) *trie.Trie { hostTrie := trie.NewTrie() for host, tag := range conf.HostMap { host = strings.ToLower(host) product := conf.HostTagMap[tag] - hostTrie.Set(strings.Split(reverseFqdnHost(host), "."), route{product: product, tag: tag}) + hostTrie.Set(strings.Split(string_reverse.ReverseFqdnHost(host), "."), route{product: product, tag: tag}) } return hostTrie diff --git a/bfe_route/server_data_conf.go b/bfe_route/server_data_conf.go index ce0e3755d..24252a23e 100644 --- a/bfe_route/server_data_conf.go +++ b/bfe_route/server_data_conf.go @@ -108,7 +108,7 @@ func (s *ServerDataConf) clusterTableLoad(clusterConf string) error { func (s *ServerDataConf) check() error { // check product consistency in host and route - for product1 := range s.HostTable.productRouteTable { + for product1 := range s.HostTable.productAdvancedRouteTable { find := false for _, product2 := range s.HostTable.hostTagTable { if product1 == product2 { @@ -121,11 +121,38 @@ func (s *ServerDataConf) check() error { } } - // check cluster_name consistency in route and cluster_conf - for _, routeRules := range s.HostTable.productRouteTable { + for product1 := range s.HostTable.productBasicRouteTree { + find := false + for _, product2 := range s.HostTable.hostTagTable { + if product1 == product2 { + find = true + break + } + } + if !find { + return fmt.Errorf("product[%s] in route should exist in host!", product1) + } + } + + // check cluster_name of advanced rule in route and cluster_conf + for _, routeRules := range s.HostTable.productAdvancedRouteTable { + for _, routeRule := range routeRules { + if _, err := s.ClusterTable.Lookup(routeRule.ClusterName); err != nil { + return fmt.Errorf("cluster[%s] in advanced route should exist in cluster_conf", + routeRule.ClusterName) + } + } + } + + // check cluster_name of basic rule table in route and cluster_conf + for _, routeRules := range s.HostTable.productBasicRouteTable { for _, routeRule := range routeRules { + if routeRule.ClusterName == route_rule_conf.AdvancedMode { + continue + } + if _, err := s.ClusterTable.Lookup(routeRule.ClusterName); err != nil { - return fmt.Errorf("cluster[%s] in route should exist in cluster_conf", + return fmt.Errorf("cluster[%s] in basic route should exist in cluster_conf", routeRule.ClusterName) } } diff --git a/bfe_server/bfe_server_init.go b/bfe_server/bfe_server_init.go index 4df70b498..be2c82306 100644 --- a/bfe_server/bfe_server_init.go +++ b/bfe_server/bfe_server_init.go @@ -26,6 +26,8 @@ import ( ) func StartUp(cfg bfe_conf.BfeConfig, version string, confRoot string) error { + var err error + // set all available modules bfe_modules.SetModules() @@ -33,22 +35,19 @@ func StartUp(cfg bfe_conf.BfeConfig, version string, confRoot string) error { bfeServer := NewBfeServer(cfg, confRoot, version) // initial http - err := bfeServer.InitHttp() - if err != nil { + if err = bfeServer.InitHttp(); err != nil { log.Logger.Error("StartUp(): InitHttp():%s", err.Error()) return err } // initial https - err = bfeServer.InitHttps() - if err != nil { + if err = bfeServer.InitHttps(); err != nil { log.Logger.Error("StartUp(): InitHttps():%s", err.Error()) return err } // load data - err = bfeServer.InitDataLoad() - if err != nil { + if err = bfeServer.InitDataLoad(); err != nil { log.Logger.Error("StartUp(): bfeServer.InitDataLoad():%s", err.Error()) return err @@ -61,22 +60,19 @@ func StartUp(cfg bfe_conf.BfeConfig, version string, confRoot string) error { // init web monitor monitorPort := cfg.Server.MonitorPort - err = bfeServer.InitWebMonitor(monitorPort) - if err != nil { + if err = bfeServer.InitWebMonitor(monitorPort); err != nil { log.Logger.Error("StartUp(): InitWebMonitor():%s", err.Error()) return err } // register modules - err = bfeServer.RegisterModules(cfg.Server.Modules) - if err != nil { + if err = bfeServer.RegisterModules(cfg.Server.Modules); err != nil { log.Logger.Error("StartUp(): RegisterModules():%s", err.Error()) return err } // initialize modules - err = bfeServer.InitModules() - if err != nil { + if err = bfeServer.InitModules(); err != nil { log.Logger.Error("StartUp(): bfeServer.InitModules():%s", err.Error()) return err @@ -84,15 +80,13 @@ func StartUp(cfg bfe_conf.BfeConfig, version string, confRoot string) error { log.Logger.Info("StartUp():bfeServer.InitModules() OK") // load plugins - err = bfeServer.LoadPlugins(cfg.Server.Plugins) - if err != nil { + if err = bfeServer.LoadPlugins(cfg.Server.Plugins); err != nil { log.Logger.Error("StartUp():bfeServer.LoadPlugins():%s", err.Error()) return err } // initialize plugins - err = bfeServer.InitPlugins() - if err != nil { + if err = bfeServer.InitPlugins(); err != nil { log.Logger.Error("StartUp():bfeServer.InitPlugins():%s", err.Error()) return err @@ -100,8 +94,7 @@ func StartUp(cfg bfe_conf.BfeConfig, version string, confRoot string) error { log.Logger.Info("StartUp():bfeServer.InitPlugins() OK") // initialize listeners - err = bfeServer.InitListeners(cfg) - if err != nil { + if err = bfeServer.InitListeners(cfg); err != nil { log.Logger.Error("StartUp(): InitListeners():%v", err) return err } diff --git a/bfe_tls/conn.go b/bfe_tls/conn.go index e9c756d20..57d806706 100644 --- a/bfe_tls/conn.go +++ b/bfe_tls/conn.go @@ -950,7 +950,7 @@ func (c *Conn) choosePlaintextSize() int { func (c *Conn) writeRecord(typ recordType, data []byte) (n int, err error) { // choose appropriate size for record // Note: Some IE browsers fail to parse fragmented TLS/SSL handshake message, - // we just choose dynamic record size for applicate data message. For more informaction, + // we just choose dynamic record size for application data message. For more information, // see https://support.microsoft.com/en-us/kb/2541763 plaintextSize := maxPlaintext if typ == recordTypeApplicationData { diff --git a/bfe_tls/handshake_messages.go b/bfe_tls/handshake_messages.go index fb7e1ab23..7e962a209 100644 --- a/bfe_tls/handshake_messages.go +++ b/bfe_tls/handshake_messages.go @@ -68,10 +68,24 @@ func (m *clientHelloMsg) equal(i interface{}) bool { eqStrings(m.alpnProtocols, m1.alpnProtocols) } +// GREASE reserves a set of TLS protocol values that may be advertised to ensure +// peers correctly handle unknown values. +// See: https://datatracker.ietf.org/doc/html/rfc8701#section-2 +func isGreaseVal(val uint16) bool { + return val&0x0F0F == 0x0A0A +} + // JA3String returns a JA3 fingerprint string for TLS client. // For more information, see https://github.com/salesforce/ja3 func (m *clientHelloMsg) JA3String() string { var buf bytes.Buffer + + // The client may advertise one or more GREASE values of cipher suite/extension + // named group/signature algorithm/version/PskKeyExchangeMode/ALPN identifiers. + // JA3 should ignores these values completely to ensure that programs utilizing + // GREASE can still be identified with a single JA3 hash. + // See: https://datatracker.ietf.org/doc/html/rfc8701#section-3.1 + // version fmt.Fprintf(&buf, "%d,", m.vers) // cipher surites @@ -81,11 +95,16 @@ func (m *clientHelloMsg) JA3String() string { writeJA3Uint16Values(&buf, m.extensionIds) fmt.Fprintf(&buf, ",") // elliptic curves - for i, curve := range m.supportedCurves { - fmt.Fprintf(&buf, "%d", curve) - if i != len(m.supportedCurves)-1 { + dashFlag := false + for _, curve := range m.supportedCurves { + if isGreaseVal(uint16(curve)) { + continue + } + if dashFlag { fmt.Fprintf(&buf, "-") } + fmt.Fprintf(&buf, "%d", curve) + dashFlag = true } fmt.Fprintf(&buf, ",") // elliptic curves point formats @@ -99,11 +118,16 @@ func (m *clientHelloMsg) JA3String() string { } func writeJA3Uint16Values(buf *bytes.Buffer, values []uint16) { - for i, value := range values { - fmt.Fprintf(buf, "%d", value) - if i != len(values)-1 { + dashFlag := false + for _, value := range values { + if isGreaseVal(value) { + continue + } + if dashFlag { fmt.Fprintf(buf, "-") } + fmt.Fprintf(buf, "%d", value) + dashFlag = true } } diff --git a/bfe_tls/handshake_messages_test.go b/bfe_tls/handshake_messages_test.go index 04eb208fb..6e8676343 100644 --- a/bfe_tls/handshake_messages_test.go +++ b/bfe_tls/handshake_messages_test.go @@ -280,6 +280,16 @@ var ja3HashTests = []struct { {769, []uint16{4, 5, 10, 9, 100, 98, 3, 6, 19, 18, 99}, []uint16{}, []CurveID{}, []uint8{}, "de350869b8c85de67a350c8d186f11e6"}, + {771, []uint16{56026, 4865, 4866, 4867, 49195, 49199, 49196, 49200, 52393, 52392, 49171, 49172, 156, 157, 47, 53}, + []uint16{60138, 0, 23, 65281, 10, 11, 35, 16, 5, 13, 18, 51, 45, 43, 27, 17513, 56026, 21}, + []CurveID{10794, 29, 23, 24}, + []uint8{0}, + "cd08e31494f9531f560d64c695473da9"}, + {771, []uint16{4865, 4866, 4867, 49195, 49199, 49196, 49200, 52393, 52392, 49171, 49172, 156, 157, 47, 53}, + []uint16{0, 23, 65281, 10, 11, 35, 16, 5, 13, 18, 51, 45, 43, 27, 17513, 56026, 21}, + []CurveID{29, 23, 24}, + []uint8{0}, + "cd08e31494f9531f560d64c695473da9"}, } func TestJA3Hash(t *testing.T) { diff --git a/bfe_util/string_reverse/string_reverse.go b/bfe_util/string_reverse/string_reverse.go new file mode 100644 index 000000000..cf1a360d4 --- /dev/null +++ b/bfe_util/string_reverse/string_reverse.go @@ -0,0 +1,30 @@ +// Copyright (c) 2019 The BFE Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package string_reverse + +// ReverseFqdnHost reverse host. +// i.e.: www.baidu.com news.baidu.com -> moc.udiab.swen moc.udiab.www will have same prefix +func ReverseFqdnHost(host string) string { + r := []rune(host) + for i, j := 0, len(r)-1; i < j; i, j = i+1, j-1 { + r[i], r[j] = r[j], r[i] + } + + if len(r) > 0 && r[0] == '.' { + r = r[1:] + } + + return string(r) +} diff --git a/bfe_websocket/server_conn.go b/bfe_websocket/server_conn.go index 7dc3ba282..77998100d 100644 --- a/bfe_websocket/server_conn.go +++ b/bfe_websocket/server_conn.go @@ -39,7 +39,7 @@ type serverConn struct { srv *Server // server config for websocket proxy hs *http.Server // server config for http req *http.Request // handshake request - rw http.ResponseWriter // for hanshake response + rw http.ResponseWriter // for handshake response cconn net.Conn // underlying conn to client bconn net.Conn // underlying conn to backend bbr *bufio.Reader // buffer reader to backend diff --git a/docs/en_us/COMMUNITY.md b/docs/en_us/COMMUNITY.md index 3c9314b5b..5e6be5468 100644 --- a/docs/en_us/COMMUNITY.md +++ b/docs/en_us/COMMUNITY.md @@ -2,7 +2,9 @@ We are very interested in building a community around BFE. If you are interested ## COMMUNITY -**BFE community on Slack**: [Sign up](https://join.slack.com/t/bfe-networks/shared_invite/zt-cn04xsqr-j7LDFmPkCuCZ39OLcHlMBA) and join channels on topics that interest you. +**BFE user forum**: [Github Discussions](https://github.com/bfenetworks/bfe/discussions). + +**BFE community on Slack**: [Sign up](https://slack.cncf.io/) CNCF Slack and join bfe channel. **WeChat developer group**: [Send a request mail](mailto:bfe-osc@baidu.com) with your WeChat ID and a contribution you have made to BFE(such as a PR/Issue). We will invite you right away. diff --git a/docs/en_us/DOWNLOAD.md b/docs/en_us/DOWNLOAD.md index c677dfe36..d6016d81f 100644 --- a/docs/en_us/DOWNLOAD.md +++ b/docs/en_us/DOWNLOAD.md @@ -1,5 +1,17 @@ We provide precompiled binaries for bfe components. [Download the latest release](https://github.com/bfenetworks/bfe/releases) of BFE for your platform. +## bfe v1.2.0 + +* 2021-06-21 [Release notes](https://github.com/bfenetworks/bfe/releases/tag/v1.2.0) + +| File name | OS | Arch | Size | SHA256 Checksum | +| --------- | -- | ---- | ---- | --------------- | +| [bfe_1.2.0_darwin_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.2.0/bfe_1.2.0_darwin_amd64.tar.gz) | darwin | amd64 | 11.9 MB | 2ebd507dbc469bba3bd3600523aa6c7c4cd306249a015f3af9fe110445243398 | +| [bfe_1.2.0_linux_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.2.0/bfe_1.2.0_linux_amd64.tar.gz) | linux | amd64 | 12.9 MB | 410eb77e963adeaf0892639d1dfd9ac048027a2fba02f5efc1374aced4134809 | +| [bfe_1.2.0_linux_arm64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.2.0/bfe_1.2.0_linux_arm64.tar.gz) | linux | arm64 | 11.8 MB | 87c83da7e182fe556f60be951c7c611f9ae144fa04d87986a5c18bcd93d9dde9 | +| [bfe_1.2.0_windows_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.2.0/bfe_1.2.0_windows_amd64.tar.gz) | windows | amd64 | 12.1 MB | e1b920fd6d8a4454120822e1640d2fd65c90fd7ec77983a661f426c82918cecd | + + ## bfe v1.1.0 * 2021-04-08 [Release notes](https://github.com/bfenetworks/bfe/releases/tag/v1.1.0) diff --git a/docs/en_us/introduction/route.md b/docs/en_us/introduction/route.md index 36f773cd2..e6726aeca 100644 --- a/docs/en_us/introduction/route.md +++ b/docs/en_us/introduction/route.md @@ -2,31 +2,237 @@ ## Overview -- In BFE [forwarding model](./forward_model.md), after "product name" for one request is determined, the destionation cluster should be identified. -- A Forwarding Table is provided for each product inside BFE. -- For each request, the forwarding table for the determined product is searched, and destination cluster is identified. +- In BFE [forwarding model](./forward_model.md), after "product name" for one request is determined, the destination cluster should be specified. +- BFE provides a Forwarding Table for each product. +- For each request, the forwarding table for the determined product is searched, and destination cluster is specified. -## Forwarding Table +## Composition of Forwarding Table -- Forwarding table is composed by one or more "forwarding rules". -- Each forwarding rule has two parts: condition(for matching the request), destionation cluster. -- Condition is expressed in [Condition Expression](../condition). -- For a request, multiple rules in forwarding table are searched up to down.(i.e., the order of forwarding rules is very important.) The procedure will stop if some rule is matched. -- There must be one Default Rule in the forwarding table. If no other rules is matched for a request, the destionation cluster defined in Default Rule is returned. +A Forwarding Table is composed of two parts: +- **Basic Rule Table**, consists of "Basic Rules". BFE searches "host" and/or "path" conditions to match a Basic Rule. +- **Advanced Rule Table**, consists of "Advanced Rules". It's searched in listed order (from up to down). Various request contents (including host, path, header, cookie, method, etc) can be used in search to match conditions in advanced rules. A Default Rule should be configured in Advanced Rule Table, as the last resort. -## Example +Basic Rule Table uses tree search and can be searched fast, even with large amount (thousands, for example) of rules configured. Advanced Rule Table has more powerful condition description capability, but its search performance may degrades when number of rules is large ( >100 rules, for example). -- A product "demo" has three clusters: - - demo-static：serve static content - - demo-post：serve post request - - demo-main: serve other traffic +## Steps of search -- The expected scenarios: - - Requests with path prefixed with "/static" will be forwarded to cluster demo-static. - - Request using method "POST" and prefixed with "/setting" will be forwarded to cluster "demo-post". - - Other request will be forwarded to cluster "demo-main". +- BFE will search in Basic Rule Table first, then Advanced Rule Table, to specify the destination cluster for a request. +- If required, destination cluster of a Basic Rule can be set to "ADVANCED_MODE". If such a rule is hit, BFE will continue to search in the Advanced Rule Table for this request. +- Detailed steps of search are described as below. -- The corresponding forwarding table is shown as follows. +![route table](../../images/route-tables-en.png) + +## Basic Rule Table + +### Introduction + +**Basic Rule Table** consists of several "Basic Rules". + +Each Basic Rule includes one or both of two conditions: Host and Path. + +Destination Cluster is specified by a cluster name, or keyword "ADVANCED_MODE" which indicates to continue searching in Advanced Rule Table. + +There's no order among Basic Rules. The matched rule with most specific condition will take precedence. See details in "Search in Basic Rule Table". + +### Conditions of Basic Rule Table + +For each Basic Rule, at least one of two conditions (Host and Path) should be configured. + +**Host** condition description's syntax is as follow: + +- Use "." to split labels within a host name +- Support "Exact Match", "Wildcard Match", "Any Match" + - Exact Match: An exact hostname (for example "www.test1.com") + - Wildcard Match: A host name with first label set to "\*". The "\*" can only appear once in a hostname and only covers a single label (for example "\*.test1.com"). Examples of invalid host condition description include "\*est.com" and "\*.\*.com". + - Any Match: A special Wildcard Match. Standalone "\*" can match any host. (A standalone "\*" here can cover a hostname with multiple labels, which is different from Wildcard Match.) +- Host condition of a Basic Rule supports multiple host condition descriptions (for example: "www.test1.com,","\*.example.com"). + +**Examples:** + +| Host Condition | Host in a request | Match？ | +| -------------- | ------------------ | ----------------------------------------- | +| \* | www.test1.com | Match | +| \*.test1.com | host.test1.com | Match | +| \*.test1.com | vip.host.test1.com | No Match, "\*" only covers a single label | +| \*.test1.com | example.com | No Match | +| \*.test1.com | test1.com | No Match | + +**Path** condition description's syntax is as follow: + +- Use "/" to split elements within a path +- Start with "/", except for a standalone "\*". +- Support "Exact Match", "Prefix Match", "Any Match" + - Exact Match: An exact path (for example "/foo") + - Prefix Match: A path prefix followed by a "\*" means it's a prefix match. It compares element by element from the left. The "\*" can only appear once, and can covers one or multiple consecutive path elements. A standalone "\*" is also a prefix match. Examples: + - both /\* and /foo/\* can match path /foo/bar + - /foo/* is equivalent to /foo\* + - /foo/b\* can not match /foo/bar + - /\*/\* is not a valid path condition description +- Path condition of a Basic Rule supports multiple path condition descriptions (for example: "/foo/bar", "/foo/cell/\*"). + +**Examples:** + +| Path Condition | Path in the request | Match? | +| ------------------------------ | ------------------- | --------------------------------------------------- | +| \* | Any path | Match | +| / | Empty | No match | +| / | /a | No match | +| /\* | Empty | No match | +| /\* | / | Match, \* can match null | +| /\* | /a/ | Match, ignores trailing slash | +| /a/b/\* (equivalent to /a/b\*) | /a/b/c | Match | +| /a/b/\* (equivalent to /a/b\*) | /a/b/c/d | Match, \* covers multiple consecutive path elements | +| /a/b/\* (equivalent to /a/b\*) | /a/b | Match, ignores trailing slash and \* can match null | +| /a/b/\* (equivalent to /a/b\*) | /a/c | No match | +| /a/b/\* (equivalent to /a/b\*) | /a/ | No match | +| /a/b\* | /a/bacon | No match | + +### Search in Basic Rule Table + +When search in Basic Rule Table, BFE will search the host condition first, then search the path condition in rules that match the host. + +Detailed steps of search in Basic Rule Table are described below: + +1. First, search host condition using **Exact Match** + - For rules that match (may be one or multiple rules), search path condition of them; + - If a rule matches the path, the rule is hit; + - If no rule matches path, then search in Basic Rule Table ends and BFE will continue to search in Advanced Rule Table. +2. If no rule matches as host Exact Match, search host condition using **Wildcard Match** + - For rules that match (may be one or multiple rules), search path condition of them; + - If a rule matches the path, the rule is hit; + - If no rule matches path, then search in Basic Rule Table ends and BFE will continue to search in Advanced Rule Table. +3. If no rule matches as host Wildcard Match, search host condition using **Any Match** + - For rules that match (may be one or multiple rules), search path condition of them; + - If a rule matches the path, the rule is hit; + - If no rule matches path, then search in Basic Rule Table ends and BFE will continue to search in Advanced Rule Table. +4. Search in Basic Rule Table ends. If no rule matches, BFE will continue to search in Advanced Rule Table. + +Among above steps, when searching path condition in rules that has matches the host: + +1. Search path condition using **Exact Match** +2. If no rule matches as Exact Match, search path condition using **Prefix Match**. Precedence will be given to the longest matching path. So if more than one rule matches the path in the request, the rule with most matching path elements is hit. + +### Examples + +Four Basic Rules are configured, as below: + +- Rule1：**host condition**：\*.test1.com，**path condition**：empty，**Destination Cluster**：StaticCluster +- Rule2：**host condition**：\*.b.test1.com，**path condition**：/interface/\* ，**Destination Cluster**：PhpCluster +- Rule3：**host condition**：\*.b.test1.com， **path condition**：/\*， **Destination Cluster**：StaticCluster +- Rule4：**host condition**：www.test1.com， **path condition**：/interface/d， **Destination Cluster**：PhpCluster + +A request arrives, with its URL="vip.b.test1.com/interface/d" + +BFE searches host condition first： + +1.**Exact Match** for host, no match + +2.**Wildcard Match** for host, both Rule2 and Rule3 match host (vip.b.test1.com) of the request (Notice, Rule1 does not match this host, as wildcard "\*" only covers a single label) + +Then BFE searches path condition： + +1.**Exact Match** for path, no match + +2.**Prefix Match** for path, both Rule2 and Rule3 match the path (/interface/d) of the request. And Rule2 has precedence as it has more matching path elements than Rule3. + +Search ends and Rule2 is hit. Request will be forwarded to cluster PhpCluster as specified in Rule2. + +## Advanced Rule Table + +Advanced Rule Table consists of one or more "Advanced Rules" which have an order. + +- Condition of a Advanced Rule is described using a pseudocode description called "[Condition Expression](https://www.bfe-networks.net/en_us/condition/condition_grammar/)". +- Destination Cluster is specified by a cluster name. + +When searching in Advanced Rule Table, the rules are searched from up to down, in listed order: + +- Try to match the condition of the rule with information in the HTTP request (such as host，path，query，cookie，method). If it matches, the rule is hit. +- If a rule is hit, the search stops. +- A Default Rule must be configured in the Advanced Rule Table. If no other rule matches a request, the Default Rule is hit. + +## Examples + +- A product "demo" has several clusters: Demo-A, Demo-B, Demo-C, Demo-D，Demo-E + +- The expected route is as below: + + Requests with host=www.a.com and path="/a/\*" (except "/a/b"), forwarded to Demo-A + + Requests with host=www.a.com and path="/a/b", forwarded to Demo-B + + Other requests with host=\*.a.com, forwarded to Demo-C + + Requests with host=www.c.com, forwarded to Demo-D + + For Demo-D, another cluster Demo-D1 is created for a canary release. For requests with host=www.c.com and cookie "deviceid" with its value starting with "x", forwarded to cluster Demo-D1 + + All the other requests, forward to Demo-E + +- In this case, the **Basic Rule Table** can be configured as below: + +| Host condition | Path condition | Destination Cluster | +| -------------- | -------------- | ------------------- | +| www.a.com | /a/\* | Demo-A | +| www.a.com | /a/b | Demo-B | +| \*.a.com | * | Demo-C | +| www.c.com | * | ADVANCED_MODE | + +There's no order for Basic Rules. Refer to "Search in Basic Rule Table" above. + +As cookie information is need for the canary release of applications on Demo-D, the rule uses ADVANCED_MODE to do further search in Advanced Rule Table for related requests. If canary release is not required, the Destination Cluster of this rule can be set to Demo-D. + +- The **Advanced Rule Table** can be configured as below: + +| Conditions | Destination Cluster | +| ------------------------------------------------------------ | ------------------- | +| req_host_in ("www.c.com") && req_cookie_value_prefix_in ("deviceid", "x", false) | Demo-D1 | +| req_host_in ("www.c.com") | Demo-D | +| default | Demo-E | + +Advanced Rules are searched from up to down. Rule for Demo-D1 should be placed before rule for Demo-D. + +Default Rule is configured in Advanced Rule Table. Requests that do not hit any other rule will be forwarded to Demo-E. + +For above configuration, configuration file (/conf/server_data_conf/route_rule.conf) is as follows: + +``` +{ + "Version": "1.0", + "BasicRule": { + "demo": [ + { + "Hostname": ["www.a.com"], + "Path": ["/a/*"], + "ClusterName": "Demo-A" + }, + { + "Hostname": ["www.a.com"], + "Path": ["/a/b"], + "ClusterName": "Demo-B" + }, + { + "Hostname": ["*.a.com"], + "Path": "*", + "ClusterName": "Demo-C" + }, + { + "Hostname": ["www.c.com"], + "Path": "*", + "ClusterName": "ADVANCED_MODE" + } + ] + }, + "ProductRule": { + "demo": [ + { + "Cond": " req_host_in(\"www.c.com\") && req_cookie_value_prefix_in(\"deviceid\", \"x\", false)", + "ClusterName": "Demo-D1" + }, + { + "Cond": " req_host_in(\"www.c.com\")", + "ClusterName": "Demo-D" + }, + { + "Cond": "default_t()", + "ClusterName": "Demo-E" + } + ] + } +} +``` -![Forwarding Table](../../images/bfe-forwarding-table.png) diff --git a/docs/en_us/modules/mod_auth_request/mod_auth_request.md b/docs/en_us/modules/mod_auth_request/mod_auth_request.md new file mode 100644 index 000000000..39bb28268 --- /dev/null +++ b/docs/en_us/modules/mod_auth_request/mod_auth_request.md @@ -0,0 +1,88 @@ +# mod_auth_request + +## Introduction + +mod_auth_request supports sending request to the specified service for authentication. + +## Module Configuration + +### Description +conf/mod_auth_request/mod_auth_request.conf + +| Config Item | Description | +| ----------------- | ------------------------------------------------------ | +| Basic.DataPath | String
Path of rule configuration | +| Basic.AuthAddress | String
Address of authentication service | +| Basic.AuthTimeout | Number
Timeout for authentication | +| Log.OpenDebug | Boolean
Whether enable debug log
Default False | + +### Example + +```ini +[Basic] +DataPath = mod_auth_request/auth_request_rule.data +AuthAddress = http://127.0.0.1 +AuthTimeout = 100 + +[Log] +OpenDebug = false +``` + +## Rule Configuration + +### Description +| Config Item | Description | +| ------------------ | ------------------------------------------------------------ | +| Version | String
Version of config file | +| Config | Object
Request auth rules for each product | +| Config{k} | String
Product name | +| Config{v} | Object
A list of request auth rules | +| Config{v}[] | Object
A request auth rule | +| Config{v}[].Cond | String
Condition expression, See [Condition](../../condition/condition_grammar.md) | +| Config{v}[].Enable | Boolean
Whether enable request auth rule | + +### Example +```json +{ + "Config": { + "example_product": [ + { + "Cond": "req_path_in(\"/auth_request\", false)", + "Enable": true + } + ] + }, + Version": "20190101000000" +} +``` + +For example_product, for request to path /auth_request (e.g., www.example.com/auth_request), BFE will create a request and send it to http://127.0.0.1 for authentication. + +### Actions + +| Action | Condition | +| ------ | ------------------------------------ | +| Forbid | Response status code is 401 or 403 | +| Pass | Response status code is 200 or other | + +## Metrics + +| Metric | Description | +| ------------------------- | -------------------------------- | +| AUTH_REQUEST_CHECKED | Counter for checked request | +| AUTH_REQUEST_PASS | Counter for passed request | +| AUTH_REQUEST_FORBIDDEN | Counter for forbidden request | +| AUTH_REQUEST_UNAUTHORIZED | Counter for unauthorized request | +| AUTH_REQUEST_FAIL | Counter for failed request | +| AUTH_REQUEST_UNCERTAIN | Counter for uncertain request | + +## Illustration of how BFE create auth request + +* Method: Request Method of HTTP Request created by BFE is **GET** +* Header: The request header created by the BFE is **originated from the original request**, but BFE makes following changes to the request: + * Delete following headers: Content-Length/Connection/Keep-Alive/Proxy-Authenticate/Proxy-Authorization/Te/Trailers/Transfer-Encoding/Upgrade + * Add following headers: X-Forwarded-Method(Original Request Method）、X-Forwarded-Uri（Original Request URI） +* Body: Body of HTTP Request created by BFE is **null** + + + diff --git a/docs/images/qrcode_for_gh.jpg b/docs/images/qrcode_for_gh.jpg new file mode 100644 index 000000000..f17ebad4f Binary files /dev/null and b/docs/images/qrcode_for_gh.jpg differ diff --git a/docs/images/route-tables-cn.png b/docs/images/route-tables-cn.png new file mode 100644 index 000000000..d071bb89e Binary files /dev/null and b/docs/images/route-tables-cn.png differ diff --git a/docs/images/route-tables-en.png b/docs/images/route-tables-en.png new file mode 100644 index 000000000..e579c97d1 Binary files /dev/null and b/docs/images/route-tables-en.png differ diff --git a/docs/mkdocs_en.yml b/docs/mkdocs_en.yml index 1787b46a1..0c813e819 100644 --- a/docs/mkdocs_en.yml +++ b/docs/mkdocs_en.yml @@ -98,6 +98,7 @@ nav: - 'mod_access': 'modules/mod_access/mod_access.md' - 'mod_auth_basic': 'modules/mod_auth_basic/mod_auth_basic.md' - 'mod_auth_jwt': 'modules/mod_auth_jwt/mod_auth_jwt.md' + - 'mod_auth_request': 'modules/mod_auth_request/mod_auth_request.md' - 'mod_block': 'modules/mod_block/mod_block.md' - 'mod_compress': 'modules/mod_compress/mod_compress.md' - 'mod_doh': 'modules/mod_doh/mod_doh.md' diff --git a/docs/mkdocs_zh.yml b/docs/mkdocs_zh.yml index 0f046c23d..06c850d7f 100644 --- a/docs/mkdocs_zh.yml +++ b/docs/mkdocs_zh.yml @@ -98,6 +98,7 @@ nav: - 'mod_access': 'modules/mod_access/mod_access.md' - 'mod_auth_basic': 'modules/mod_auth_basic/mod_auth_basic.md' - 'mod_auth_jwt': 'modules/mod_auth_jwt/mod_auth_jwt.md' + - 'mod_auth_request': 'modules/mod_auth_request/mod_auth_request.md' - 'mod_block': 'modules/mod_block/mod_block.md' - 'mod_compress': 'modules/mod_compress/mod_compress.md' - 'mod_doh': 'modules/mod_doh/mod_doh.md' diff --git a/docs/zh_cn/COMMUNITY.md b/docs/zh_cn/COMMUNITY.md index 7bb0453ad..82a0cd2d7 100644 --- a/docs/zh_cn/COMMUNITY.md +++ b/docs/zh_cn/COMMUNITY.md @@ -2,10 +2,12 @@ ## BFE社区 -**BFE Slack社区**: [登陆](https://join.slack.com/t/bfe-networks/shared_invite/zt-cn04xsqr-j7LDFmPkCuCZ39OLcHlMBA)并关注你感兴趣的话题。 +**BFE用户论坛**: 前往[Github Discussions](https://github.com/bfenetworks/bfe/discussions)讨论。 **BFE开发者微信群**: [发送申请邮件](mailto:bfe-osc@baidu.com)并注明您的微信账号及贡献(例如提交的PR/Issue链接)，我们将立即邀请您加入。 +**BFE Slack社区**: [登陆](https://slack.cncf.io/)CNCF Slack并关注bfe频道。 + **Twitter**: [@BFE-Networks](https://twitter.com/BfeNetworks) **Issue tracker**: 在[GitHub issue tracker](https://github.com/bfenetworks/bfe/issues) 反馈bug或功能需求。 diff --git a/docs/zh_cn/DOWNLOAD.md b/docs/zh_cn/DOWNLOAD.md index a90db79ef..28704aeea 100644 --- a/docs/zh_cn/DOWNLOAD.md +++ b/docs/zh_cn/DOWNLOAD.md @@ -1,11 +1,23 @@ BFE提供预编译二进制文件供下载。也可在GitHub下载各平台[最新版本BFE](https://github.com/bfenetworks/bfe/releases)。 +## bfe v1.2.0 + +* 2021-06-21 [发布说明](https://github.com/bfenetworks/bfe/releases/tag/v1.2.0) + +| 文件名 | 操作系统 | 平台 | 大小 | SHA256检验和 | +| --------- | -------- | ---- | ---- | ------------ | +| [bfe_1.2.0_darwin_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.2.0/bfe_1.2.0_darwin_amd64.tar.gz) | darwin | amd 64 | 11.9 MB | 2ebd507dbc469bba3bd3600523aa6c7c4cd306249a015f3af9fe110445243398 | +| [bfe_1.2.0_linux_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.2.0/bfe_1.2.0_linux_amd64.tar.gz) | linux | amd64 | 12.9 MB | 410eb77e963adeaf0892639d1dfd9ac048027a2fba02f5efc1374aced4134809 | +| [bfe_1.2.0_linux_arm64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.2.0/bfe_1.2.0_linux_arm64.tar.gz) | linux | arm64 | 11.8 MB | 87c83da7e182fe556f60be951c7c611f9ae144fa04d87986a5c18bcd93d9dde9 | +| [bfe_1.2.0_windows_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.2.0/bfe_1.2.0_windows_amd64.tar.gz) | windows | amd64 | 12.1 MB | e1b920fd6d8a4454120822e1640d2fd65c90fd7ec77983a661f426c82918cecd | + + ## bfe v1.1.0 * 2021-04-08 [发布说明](https://github.com/bfenetworks/bfe/releases/tag/v1.1.0) | 文件名 | 操作系统 | 平台 | 大小 | SHA256检验和 | -| --------- | -- | ---- | ---- | --------------- | +| --------- | -------- | ---- | ---- | ------------ | | [bfe_1.1.0_darwin_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.1.0/bfe_1.1.0_darwin_amd64.tar.gz) | darwin | amd64 | 11.9 MB | 95a1cfe762008533886d6fb68b38cc9c492491216d6e39bfb8003785a366e22b | | [bfe_1.1.0_linux_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.1.0/bfe_1.1.0_linux_amd64.tar.gz) | linux | amd64 | 12.9 MB | 7dd8fc826c4a4b147b6ca2a19e713ecd9450b099f45e123498fdf63221b59cf2 | | [bfe_1.1.0_linux_arm64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.1.0/bfe_1.1.0_linux_arm64.tar.gz) | linux | arm64 | 11.8 MB | a2b6047eb4121a0ff6a0255699384a94b094f9d60553ad9e2a11ee39647320bd | @@ -16,7 +28,7 @@ BFE提供预编译二进制文件供下载。也可在GitHub下载各平台[最 * 2021-01-15 [发布说明](https://github.com/bfenetworks/bfe/releases/tag/v1.0.0) | 文件名 | 操作系统 | 平台 | 大小 | SHA256检验和 | -| --------- | -- | ---- | ---- | --------------- | +| --------- | -------- | ---- | ---- | ------------ | | [bfe_1.0.0_darwin_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.0.0/bfe_1.0.0_darwin_amd64.tar.gz) | darwin | amd64 | 7.03 MB | c0d13440d89ab97f52c61610d1b10dec6dcfb47b468a66078d1dd60f0541ec9e | | [bfe_1.0.0_linux_amd64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.0.0/bfe_1.0.0_linux_amd64.tar.gz) | linux | amd64 | 6.18 MB | 5ec46c26827d554ba4c76f7f5e12b6b6afb68a9333213065802fa425fb81cbd1 | | [bfe_1.0.0_linux_arm64.tar.gz](https://github.com/bfenetworks/bfe/releases/download/v1.0.0/bfe_1.0.0_linux_arm64.tar.gz) | linux | arm64 | 5.63 MB | 47a3730ac90c4700c557d6c5903361c557e169102256bac870cede4eb90ff829 | diff --git a/docs/zh_cn/faq/development.md b/docs/zh_cn/faq/development.md index b02dca872..4ed51b548 100644 --- a/docs/zh_cn/faq/development.md +++ b/docs/zh_cn/faq/development.md @@ -1,4 +1,4 @@ # 开发常见问题 ## 如何开发BFE扩展模块 -- 具体见[模块开发介绍](https://github.com/bfenetworks/bfe/blob/develop/docs/zh_cn/module/modules.md) +- 具体见[模块开发介绍](https://github.com/bfenetworks/bfe/blob/develop/docs/zh_cn/modules/modules.md) diff --git a/docs/zh_cn/introduction/route.md b/docs/zh_cn/introduction/route.md index e0a3fbfe2..83804336d 100644 --- a/docs/zh_cn/introduction/route.md +++ b/docs/zh_cn/introduction/route.md @@ -4,32 +4,237 @@ ## 概述 - 在BFE的[接入转发流程](./forward_model.md)中，在确定产品线后，要进一步确定处理该请求的目标集群 -- 在BFE内，为每个产品线维护一张“转发表” +- BFE为每个产品线维护一张“转发表” - 对每个属于该产品线的请求，查询转发表，获得目标集群 -## 转发表 +## 转发表的组成 -- 转发表由多条“转发规则”组成 -- 每条转发规则包括两部分：匹配条件，目标集群 -- 匹配条件使用“[条件表达式（Condition Expression）](../condition)”来表述 -- 多条转发规则顺序执行。只要命中任何一条转发规则，就会结束退出 -- 转发表必须包含“默认规则（Default）”。在所有转发规则都没有命中的时候，执行默认规则 +转发表包括2部分： + +- **基础规则表**：由基础规则组成。使用域名(Host)和路径(Path)作为匹配条件，基础规则表中的规则按最长匹配原则进行匹配。 +- **高级规则表**：由高级规则组成。使用请求中的多种信息（如host、path、header、cookie、method等）进行匹配，高级规则表中的规则按先后顺序进行顺序匹配。在高级规则表中可以设置默认规则。 + +基础规则表使用树形查找，匹配速度快，可以支持较大数量（几千甚至上万）转发规则的快速查找；高级规则表支持更强大的条件描述能力，但是在规则数量较多（超过100条）时会有明显的性能下降。 + +## 匹配顺序 + +BFE会按照基础规则表、高级规则表的顺序来查找，以确定目标集群。 + +在基础规则表中，也可以将某条规则的目标集群设置为“ADVANCED_MODE”(高级模式)，命中此规则的请求将转至高级规则表进一步匹配。 + +详细的匹配顺序见下图。 + +![route table](../../images/route-tables-cn.png) + + + +## 基础规则表 + +### 基础规则说明 + +基础规则表由多条“基础规则”组成。 + +- 基础规则的匹配条件包括域名(host)和路径(path)两个部分。 +- 目的集群通过集群名称来指定，也可以设置为“ADVANCED_MODE”转至高级规则表继续匹配。 + +基础规则没有前后顺序关系，而是基于精确优先、最长匹配的原则。详细的匹配方式见后面的“基础规则匹配顺序”。 + +### 基础规则匹配条件 + +一条基础规则可以指定host条件或path条件，或者同时指定host条件和path条件。 + +host条件的描述语法遵循以下规则： + +- 以"."作为域名标签元素的分割符（域名标签元素指的是由 "."分隔的host中的标签。） +- 支持“精确匹配”、“通配符匹配”和“任意匹配” + - 精确匹配：完整的域名，如：www.test1.com + - 通配符匹配：第一个域名标签元素为通配符"\*"表示这个描述为通配符匹配。 "\*"仅能出现一次，且仅可以表示一个域名标签元素，例如\*.test.com。而\*est.com和\*.\*.com均不符合语法。 + - 任意匹配：一种特殊的通配符匹配，可以填写单独的通配符"\*"来表示匹配任意的host值（与通配符匹配不同，任意匹配的\*可以匹配包含多个域名标签元素的host） +- 一条规则的host条件，支持多个host描述，如："www.test1.com", "\*.example.com" + +示例： + +| host描述 | 请求的host | 是否匹配？ | +| ----------- | -------------------------------------- | ------------------------------- | +| \* | www.test1.com | 匹配 | +| \*.test1.com | host.test1.com | 匹配 | +| \*.test1.com | vip.host.test1.com | 不匹配，\* 只匹配一个域名标签元素 | +| \*.test1.com | example.com | 不匹配 | +| \*.test1.com | test1.com | 不匹配 | + +path条件的描述语法遵循以下规则： + +- 以"/"作为路径元素的分割符（路径元素指的是由 "/" 分隔符分隔的路径中的标签。） +- 除了任意匹配，均以"/"开始 +- 支持“精确匹配”和“前缀匹配” + - 精确匹配：完整的路径，如/path + - 前缀匹配：在路径前缀后面使用"\*"来表示这个描述为前缀匹配，即从左开始对path中的路径元素逐个匹配。 "\*"仅能出现一次，且可以表示一个或多个连续的路径元素。前缀匹配的描述也可以是一个单独的"\*"。举例如下： + - /foo/bar 这个path，条件 /\* 和 /foo/\* 均能匹配到； + - /foo/\* 和 /foo\* 是等价的； + - /foo/bar 这个path，不能被条件 /foo/b\* 匹配到； + - /\*/\* 不符合path描述的语法 +- 一条规则的path条件，支持多个path描述，如："/foo/bar", "/foo/cell/\*" + +更多示例： + +| path描述 | 请求的path | 是否匹配 | +| ----------- | ---------- | ----------------------------------------- | +| \* | 任何path值 | 匹配 | +| / | 空值 | 不匹配 | +| / | /a | 不匹配 | +| /\* | 空值 | 不匹配 | +| /\* | / | 匹配，\*可以匹配空值 | +| /\* | /a/ | 匹配，忽略尾部斜线 | +| /a/b/\* (与 /a/b\* 等价) | /a/b/c | 匹配 | +| /a/b/\* (与 /a/b\* 等价) | /a/b/c/d | 匹配，\*可以匹配多个路径元素 | +| /a/b/\* (与 /a/b\* 等价) | /a/b | 匹配，忽略尾部斜线，\*可以匹配空值 | +| /a/b/\* (与 /a/b\* 等价) | /a/c | 不匹配 | +| /a/b/\* (与 /a/b\* 等价) | /a | 不匹配 | +| /a/b\* | /a/bacon | 不匹配 | + +### 基础规则匹配顺序 + +在基础规则表中查找时，先host条件匹配，符合host条件的规则，再进行path条件匹配： + +1. 首先对host进行**精确匹配** + - 对匹配到的规则（可能是一条或多条），再尝试匹配这些规则的path条件； + - 若匹配path条件成功，即为命中规则； + - 若匹配path条件未成功，则判定为未命中基础规则，转由高级规则进行匹配 +2. host精确匹配未成功的，对host进行**通配符匹配** + - 对匹配到的规则（可能是一条或多条），再尝试匹配这些规则的path条件； + - 若匹配path条件成功，即为命中规则； + - 若匹配path条件未成功，则判定为未命中基础规则，转由高级规则进行匹配 +3. host通配符匹配未成功的，对host进行**任意匹配** + - 对匹配到的规则（可能是一条或多条），再尝试匹配这些规则的path条件； + - 若匹配path条件成功，即为命中规则； + - 若匹配path条件未成功，则判定为未命中基础规则，转由高级规则进行匹配 +4. 基础规则匹配结束，未命中的，转由高级规则进行匹配 + + +其中，对于上面步骤中，对host条件匹配到的规则，进行匹配path条件时的匹配逻辑如下： + +1. 首先对path进行精确匹配 +2. 精确匹配未成功的，对path进行前缀匹配。前缀匹配使用最长匹配原则，即若请求的path符合多条规则的path条件，匹配到从左开始有最多路径元素的那条规则（只会是一条规则） + +### 基础规则匹配示例 + +有四条基础规则，分别为： + +- 规则1：**host条件**：\*.test1.com，**path条件**：空值，**目标集群**：StaticCluster +- 规则2：**host条件**：\*.b.test1.com，**path条件**：/interface/\* ，**目标集群**：PhpCluster +- 规则3：**host条件**：\*.b.test1.com， **path条件**：/\*， **目标集群**：StaticCluster +- 规则4：**host条件**：www.test1.com， **path条件**：/interface/d， **目标集群**：PhpCluster + +收到一个请求，请求URL为vip.b.test1.com/interface/d + +首先对host匹配： + +1.host**精确匹配**，未匹配成功 + +2.host**通配符匹配**，发现请求的host(vip.b.test1.com)可以满足规则2，3的host条件（注意，规则1不匹配该请求，因为通配符"\*"只能匹配一个域名标签元素）。 + +继续进行path匹配： + +1.path**精确匹配**，未匹配成功 + +2.path**前缀匹配**，发现请求的path(/interface/d)可以满足规则2和规则3的path条件，根据最长匹配原则，匹配到从左开始有最多路径元素的那条规则，即规则2。 + +匹配结束，在基础规则表中命中规则2。根据规则2的设置，将请求转发到集群PhpCluster。 + +## 高级规则表 + +高级规则列表由多条**有顺序**的“高级规则”组成。 + +- 高级规则的匹配条件使用了一个伪码描述机制“[条件表达式](../condition/condition_grammar.md)”（Condition Expression，后面简称为Condition或条件表达式）来表述。 +- 目的集群通过集群名称来指定。 + +在查询时，对多条高级规则顺序查找： + +- 基于HTTP请求中的信息，根据转发规则的匹配条件进行条件判断，符合匹配条件的为命中该转发规则。可以使用的请求信息包括：host，path，query，cookie，method等 +- 只要命中任何一条转发规则，就会由此确定目的集群并结束退出。 +- 高级规则表中包含“默认规则（Default）。若所有转发规则都没有命中，将执行默认规则。 ## 示例 -- 产品线demo，包含以下3种服务集群： - + 静态集群(demo-static)：服务静态流量 - + post集群(demo-post)：服务post流量 - + main集群(demo-main)：服务其他流量 -- 期望的转发条件如下： - + 对于path以"/static"为前缀的，都发往demo-static集群 - + 请求方法为"POST"、且path以"/setting"为前缀的，都发往demo-post - + 其它请求，都发往demo-main -- 对应以上要求，产品线demo的转发表如下图所示 -![Forwarding Table](../../images/bfe-forwarding-table.png) +- 产品线demo，包含多种服务集群：Demo-A, Demo-B, Demo-C, Demo-D，Demo-E + +- 期望的集群间分流如下： + + 对于host为www.a.com，且path为"/a/\*" (除了"/a/b") 的请求，转发至Demo-A集群 + + 对于host为www.a.com，且path为"/a/b"的请求，转发至Demo-B集群 + + 对于其他host为\*.a.com的请求，转发至Demo-C集群 + + 对于host为www.c.com的请求，转发至Demo-D集群 + + 针对Demo-D集群，另外开启了一个灰度集群Demo-D1。如果cookie中包含deviceid，且这个cookie的值以“x”开头，则转发至Demo-D1 + + 其它请求，都发往Demo-E + +- 对应以上要求，**基础规则表**的配置为： + +| host条件 | path条件 | 目标集群 | +| ----------- | --------------------------- | ------------------ | +| www.a.com | /a/\* | Demo-A | +| www.a.com | /a/b | Demo-B | +| \*.a.com | \* | Demo-C | +| www.c.com | \* | ADVANCED_MODE | + +在基础规则表中，规则之间没有前后顺序。见前面关于“基础规则匹配顺序”的说明。 + +针对Demo-D集群上应用的灰度发布，由于需要使用cookie中的信息，所以使用ADVANCED_MODE将满足条件的请求透传到高级规则表继续处理；在不需要灰度发布的时候，在基础规则表中www.c.com对应规则的目标集群写为Demo-D即可。 +- **高级规则表**的配置为： +| 匹配条件 | 目标集群 | +| ---------------------------------------- | ------------------ | +| req_host_in("www.c.com") && req_cookie_value_prefix_in("deviceid", "x", false) | Demo-D1 | +| req_host_in("www.c.com") | Demo-D | +| default | Demo-E | +在高级规则表中，多条规则之间是有序的。需要将转发给Demo-D1的规则放在转发给Demo-D的规则的前面。 +高级规则表中包含默认规则，对于没有命中其它规则的请求将被转发到Demo-E。 +以上配置信息，对应的配置文件（/conf/server_data_conf/route_rule.conf）如下： +``` +{ + "Version": "1.0", + "BasicRule": { + "demo": [ + { + "Hostname": ["www.a.com"], + "Path": ["/a/*"], + "ClusterName": "Demo-A" + }, + { + "Hostname": ["www.a.com"], + "Path": ["/a/b"], + "ClusterName": "Demo-B" + }, + { + "Hostname": ["*.a.com"], + "Path": "*", + "ClusterName": "Demo-C" + }, + { + "Hostname": ["www.c.com"], + "Path": "*", + "ClusterName": "ADVANCED_MODE" + } + ] + }, + "ProductRule": { + "demo": [ + { + "Cond": " req_host_in(\"www.c.com\") && req_cookie_value_prefix_in(\"deviceid\", \"x\", false)", + "ClusterName": "Demo-D1" + }, + { + "Cond": " req_host_in(\"www.c.com\")", + "ClusterName": "Demo-D" + }, + { + "Cond": "default_t()", + "ClusterName": "Demo-E" + } + ] + } +} +``` diff --git a/docs/zh_cn/modules/mod_auth_request/mod_auth_request.md b/docs/zh_cn/modules/mod_auth_request/mod_auth_request.md new file mode 100644 index 000000000..a0fc6b0b9 --- /dev/null +++ b/docs/zh_cn/modules/mod_auth_request/mod_auth_request.md @@ -0,0 +1,85 @@ +# mod_auth_request + +## 模块简介 + +mod_auth_request支持请求发送至指定的服务进行认证。 + +## 基础配置 +### 配置描述 +模块配置文件: conf/mod_auth_request/mod_auth_request.conf + +| 配置项 | 描述 | +| ----------------- | ----------------------------------------------- | +| Basic.DataPath | String
规则配置的文件路径 | +| Basic.AuthAddress | String
认证服务的地址 | +| Basic.AuthTimeout | Number
认证超时时间
单位ms | +| Log.OpenDebug | Boolean
是否开启调试日志
默认值False | + +### 配置示例 + +```ini +[Basic] +DataPath = mod_auth_request/auth_request_rule.data +AuthAddress = http://127.0.0.1 +AuthTimeout = 100 + +[Log] +OpenDebug = false +``` + +## 规则配置 +### 配置描述 +| 配置项 | 描述 | +| ------------------ | ------------------------------------------------------------ | +| Version | String
配置文件版本 | +| Config | Object
所有产品线的请求认证规则配置 | +| Config{k} | String
产品线名称 | +| Config{v} | Object
产品线的请求认证规则表 | +| Config{v}[] | Object
请求认证规则 | +| Config{v}[].Cond | String
匹配条件, 语法详见[Condition](../../condition/condition_grammar.md) | +| Config{v}[].Enable | Boolean
是否启用规则 | + +### 配置示例 + +```josn +{ + "Config": { + "example_product": [ + { + "Cond": "req_path_in(\"/auth_request\", false)", + "Enable": true + } + ] + }, + Version": "20190101000000" +} +``` + +对于example_product产品线配置了一条规则，针对请求路径为/auth_request的请求（例如www.example.com/auth_request），BFE将构造请求发送至http://127.0.0.1进行认证。 + +### 模块动作 + +| 动作 | 条件 | +| ---- | --------------------- | +| 封禁 | 响应状态码为401或403 | +| 放行 | 响应状态码为200或其他 | + +## 监控项 + +| 监控项 | 描述 | +| ------------------------- | ------------------------ | +| AUTH_REQUEST_CHECKED | 命中基本认证规则的请求数 | +| AUTH_REQUEST_PASS | 认证成功并放行的请求数 | +| AUTH_REQUEST_FORBIDDEN | 被禁止的请求数 | +| AUTH_REQUEST_UNAUTHORIZED | 未通过认证的请求数 | +| AUTH_REQUEST_FAIL | 认证失败的请求数 | +| AUTH_REQUEST_UNCERTAIN | 认证状态不确定的请求数 | + + +## BFE构造请求的说明 + +* Method: BFE构造的请求Method为GET +* Header: BFE构造的请求Header为原请求Header，同时进行如下修改： + * 删除如下头部：Content-Length/Connection/Keep-Alive/Proxy-Authenticate/Proxy-Authorization/Te/Trailers/Transfer-Encoding/Upgrade + * 增加如下头部：X-Forwarded-Method(代表原请求Method）、X-Forwarded-Uri（代表原请求URI） +* Body: BFE构造的请求Body为空 diff --git a/go.mod b/go.mod index 8dc69eb39..88dc3978c 100644 --- a/go.mod +++ b/go.mod @@ -5,13 +5,14 @@ go 1.13 require ( github.com/abbot/go-http-auth v0.4.1-0.20181019201920-860ed7f246ff github.com/andybalholm/brotli v1.0.0 + github.com/armon/go-radix v1.0.0 github.com/asergeyev/nradix v0.0.0-20170505151046-3872ab85bb56 // indirect github.com/baidu/go-lib v0.0.0-20200819072111-21df249f5e6a github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd // indirect - github.com/dgrijalva/jwt-go v3.2.0+incompatible + github.com/golang-jwt/jwt v3.2.2+incompatible github.com/gomodule/redigo v2.0.0+incompatible github.com/json-iterator/go v1.1.10 - github.com/microcosm-cc/bluemonday v1.0.3 + github.com/microcosm-cc/bluemonday v1.0.5 github.com/miekg/dns v1.1.29 github.com/opentracing/opentracing-go v1.1.0 github.com/openzipkin-contrib/zipkin-go-opentracing v0.4.5 @@ -26,7 +27,7 @@ require ( github.com/uber/jaeger-client-go v2.22.1+incompatible github.com/uber/jaeger-lib v2.2.0+incompatible github.com/zmap/go-iptree v0.0.0-20170831022036-1948b1097e25 - go.elastic.co/apm v1.7.2 + go.elastic.co/apm v1.11.0 go.elastic.co/apm/module/apmot v1.7.2 go.uber.org/atomic v1.6.0 // indirect go.uber.org/automaxprocs v1.4.0 diff --git a/go.sum b/go.sum index c34cbefe7..ef284474e 100644 --- a/go.sum +++ b/go.sum @@ -24,8 +24,6 @@ github.com/cucumber/godog v0.8.1/go.mod h1:vSh3r/lM+psC1BPXvdkSEuNjmXfpVqrMGYAEl github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM= -github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= github.com/eapache/go-resiliency v1.1.0/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5mFgVsvEsIPBvNs= github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU= github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I= @@ -37,6 +35,8 @@ github.com/envoyproxy/go-control-plane v0.6.9/go.mod h1:SBwIajubJHhxtWwsL9s8ss4s github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/gogo/googleapis v1.1.0/go.mod h1:gf4bu3Q80BeJ6H1S1vYPm8/ELATdvryBaNFGgqEef3s= github.com/gogo/protobuf v1.2.0/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= +github.com/golang-jwt/jwt v3.2.2+incompatible h1:IfV12K8xAKAnZqdXVzCZ+TOjboZ2keLg81eXfW3O+oY= +github.com/golang-jwt/jwt v3.2.2+incompatible/go.mod h1:8pz2t5EyA70fFQQSrl6XZXzqecmYZeUEB8OUGHkxJ+I= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= @@ -68,8 +68,8 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/lyft/protoc-gen-validate v0.0.13/go.mod h1:XbGvPuh87YZc5TdIa2/I4pLk0QoUACkjt2znoq26NVQ= -github.com/microcosm-cc/bluemonday v1.0.3 h1:EjVH7OqbU219kdm8acbveoclh2zZFqPJTJw6VUlTLAQ= -github.com/microcosm-cc/bluemonday v1.0.3/go.mod h1:8iwZnFn2CDDNZ0r6UXhF4xawGvzaqzCRa1n3/lO3W2w= +github.com/microcosm-cc/bluemonday v1.0.5 h1:cF59UCKMmmUgqN1baLvqU/B1ZsMori+duLVTLpgiG3w= +github.com/microcosm-cc/bluemonday v1.0.5/go.mod h1:8iwZnFn2CDDNZ0r6UXhF4xawGvzaqzCRa1n3/lO3W2w= github.com/miekg/dns v1.1.29 h1:xHBEhR+t5RzcFJjBLJlax2daXOrTYtr9z4WdKEfWFzg= github.com/miekg/dns v1.1.29/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 h1:ZqeYNhU3OHLH3mGKHDcjJRFFRrJa6eAM5H+CtDdOsPc= @@ -124,17 +124,20 @@ github.com/uber/jaeger-client-go v2.22.1+incompatible h1:NHcubEkVbahf9t3p75TOCR8 github.com/uber/jaeger-client-go v2.22.1+incompatible/go.mod h1:WVhlPFC8FDjOFMMWRy2pZqQJSXxYSwNYOkTr/Z6d3Kk= github.com/uber/jaeger-lib v2.2.0+incompatible h1:MxZXOiR2JuoANZ3J6DE/U0kSFv/eJ/GfSYVCjK7dyaw= github.com/uber/jaeger-lib v2.2.0+incompatible/go.mod h1:ComeNDZlWwrWnDv8aPp0Ba6+uUTzImX/AauajbLI56U= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/zmap/go-iptree v0.0.0-20170831022036-1948b1097e25 h1:LRoXAcKX48QV4LV23W5ZtsG/MbJOgNUNvWiXwM0iLWw= github.com/zmap/go-iptree v0.0.0-20170831022036-1948b1097e25/go.mod h1:qOasALtPByO1Jk6LhgpNv6htPMK2QJfiGorUk57nO/U= -go.elastic.co/apm v1.7.2 h1:0nwzVIPp4PDBXSYYtN19+1W5V+sj+C25UjqxDVoKcA8= go.elastic.co/apm v1.7.2/go.mod h1:tCw6CkOJgkWnzEthFN9HUP1uL3Gjc/Ur6m7gRPLaoH0= +go.elastic.co/apm v1.11.0 h1:uJyt6nCW9880sZhfl1tB//Jy/5TadNoAd8edRUtgb3w= +go.elastic.co/apm v1.11.0/go.mod h1:qoOSi09pnzJDh5fKnfY7bPmQgl8yl2tULdOu03xhui0= go.elastic.co/apm/module/apmhttp v1.7.2 h1:2mRh7SwBuEVLmJlX+hsMdcSg9xaielCLElaPn/+i34w= go.elastic.co/apm/module/apmhttp v1.7.2/go.mod h1:sTFWiWejnhSdZv6+dMgxGec2Nxe/ZKfHfz/xtRM+cRY= go.elastic.co/apm/module/apmot v1.7.2 h1:FXvTXGvVOwc26K3llgPdxenWoPv9VdO5CQ3aAfc5lZY= go.elastic.co/apm/module/apmot v1.7.2/go.mod h1:VD2nUkebUPrP1hqIarimIEsoM9xyuK0lO83fCx6l/Z8= -go.elastic.co/fastjson v1.0.0 h1:ooXV/ABvf+tBul26jcVViPT3sBir0PvXgibYB1IQQzg= go.elastic.co/fastjson v1.0.0/go.mod h1:PmeUOMMtLHQr9ZS9J9owrAVg0FkaZDRZJEFTTGHtchs= +go.elastic.co/fastjson v1.1.0 h1:3MrGBWWVIxe/xvsbpghtkFoPciPhOCmjsR/HfwEeQR4= +go.elastic.co/fastjson v1.1.0/go.mod h1:boNGISWMjQsUPy/t6yqt2/1Wx4YNPSe+mZjlyw9vKKI= go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk= go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= go.uber.org/automaxprocs v1.4.0 h1:CpDZl6aOlLhReez+8S3eEotD7Jx0Os++lemPlMULQP0= @@ -148,6 +151,7 @@ golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHl golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs= golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -157,6 +161,7 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20201021035429-f5854403a974 h1:IX6qOQeG5uLjB/hjjwjedwfjND0hgjPMMyO1RoIXQNI= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= @@ -164,6 +169,7 @@ golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9 h1:SQFwaSi55rU7vdNs9Yr0Z324VNlrF+0wMqRXT4St8ck= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -186,10 +192,12 @@ golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBn golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200509030707-2212a7e161a5/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.1.0 h1:po9/4sTYwZU9lPhi1tOrb4hCv3qrhiQ77LZfGa2OjwY= golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=